[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:48:45 dist-test-slave-hwlj krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:48:45 dist-test-slave-hwlj krb5kdc[399](info): set up 2 sockets
Sep 26 20:48:45 dist-test-slave-hwlj krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 20:48:46.583999 392 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/krb5kdc/test-user.keytab.
Sep 26 20:48:46 dist-test-slave-hwlj krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:48:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:48:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 20:48:46.702916 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37663
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37663
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:48:46.783232 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:48:46.783437 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:48:46.783458 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:48:46.784703 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:48:46.784747 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:48:46.784762 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:48:46.784775 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:48:46.784837 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:48:46.786324 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:48:46.786593 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:48:46.786852 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:48:46.792187 412 server_base.cc:1034] running on GCE node
W20260926 20:48:46.792145 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:48:46.792135 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:48:46.792153 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:48:46.792651 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:48:46.792992 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:48:46.794171 412 hybrid_clock.cc:648] HybridClock initialized: now 1790455726794158 us; error 46 us; skew 500 ppm
Sep 26 20:48:46 dist-test-slave-hwlj krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:48:46.796801 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:48:46.797336 412 webserver.cc:460] Webserver started at http://127.0.98.62:42957/ using document root <none> and password file <none>
I20260926 20:48:46.797577 412 fs_manager.cc:359] Metadata directory not provided
I20260926 20:48:46.797646 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:48:46.797794 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:48:46.798856 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/master-0/data/instance:
uuid: "55eec3ae1fe3486f975902fb50d596de"
format_stamp: "Formatted at 2026-09-26 20:48:46 on dist-test-slave-hwlj"
server_key: "2650842f17f5266b9c4ced73cd2e9831"
server_key_iv: "86c092d418b15f636919381bae72bd6d"
server_key_version: "encryptionkey@0"
I20260926 20:48:46.799250 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/master-0/wal/instance:
uuid: "55eec3ae1fe3486f975902fb50d596de"
format_stamp: "Formatted at 2026-09-26 20:48:46 on dist-test-slave-hwlj"
server_key: "2650842f17f5266b9c4ced73cd2e9831"
server_key_iv: "86c092d418b15f636919381bae72bd6d"
server_key_version: "encryptionkey@0"
I20260926 20:48:46.800674 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:48:46.801436 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:48:46.801601 412 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 20:48:46.801707 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "55eec3ae1fe3486f975902fb50d596de"
format_stamp: "Formatted at 2026-09-26 20:48:46 on dist-test-slave-hwlj"
server_key: "2650842f17f5266b9c4ced73cd2e9831"
server_key_iv: "86c092d418b15f636919381bae72bd6d"
server_key_version: "encryptionkey@0"
I20260926 20:48:46.801779 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:48:46.813164 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:48:46.815423 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:48:46.815598 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:48:46.819334 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37663
I20260926 20:48:46.819365 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37663 every 8 connection(s)
I20260926 20:48:46.819726 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:48:46.820643 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:48:46.822840 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55eec3ae1fe3486f975902fb50d596de: Bootstrap starting.
I20260926 20:48:46.823438 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55eec3ae1fe3486f975902fb50d596de: Neither blocks nor log segments found. Creating new log.
I20260926 20:48:46.823730 480 log.cc:824] T 00000000000000000000000000000000 P 55eec3ae1fe3486f975902fb50d596de: Log is configured to *not* fsync() on all Append() calls
I20260926 20:48:46.824513 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55eec3ae1fe3486f975902fb50d596de: No bootstrap required, opened a new log
I20260926 20:48:46.826085 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55eec3ae1fe3486f975902fb50d596de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55eec3ae1fe3486f975902fb50d596de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37663 } }
I20260926 20:48:46.826246 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55eec3ae1fe3486f975902fb50d596de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:48:46.826272 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55eec3ae1fe3486f975902fb50d596de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55eec3ae1fe3486f975902fb50d596de, State: Initialized, Role: FOLLOWER
I20260926 20:48:46.826440 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55eec3ae1fe3486f975902fb50d596de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55eec3ae1fe3486f975902fb50d596de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37663 } }
I20260926 20:48:46.826504 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55eec3ae1fe3486f975902fb50d596de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:48:46.826525 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55eec3ae1fe3486f975902fb50d596de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:48:46.826597 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55eec3ae1fe3486f975902fb50d596de [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:48:46.827112 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55eec3ae1fe3486f975902fb50d596de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55eec3ae1fe3486f975902fb50d596de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37663 } }
I20260926 20:48:46.827232 480 leader_election.cc:302] T 00000000000000000000000000000000 P 55eec3ae1fe3486f975902fb50d596de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55eec3ae1fe3486f975902fb50d596de; no voters:
I20260926 20:48:46.827407 480 leader_election.cc:288] T 00000000000000000000000000000000 P 55eec3ae1fe3486f975902fb50d596de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:48:46.827493 483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55eec3ae1fe3486f975902fb50d596de [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:48:46.827639 483 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55eec3ae1fe3486f975902fb50d596de [term 1 LEADER]: Becoming Leader. State: Replica: 55eec3ae1fe3486f975902fb50d596de, State: Running, Role: LEADER
I20260926 20:48:46.827656 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55eec3ae1fe3486f975902fb50d596de [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:48:46.827792 483 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55eec3ae1fe3486f975902fb50d596de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55eec3ae1fe3486f975902fb50d596de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37663 } }
I20260926 20:48:46.827818 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 412
I20260926 20:48:46.827911 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/master-0/wal/instance
I20260926 20:48:46.828289 484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55eec3ae1fe3486f975902fb50d596de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55eec3ae1fe3486f975902fb50d596de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55eec3ae1fe3486f975902fb50d596de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37663 } } }
I20260926 20:48:46.828361 484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55eec3ae1fe3486f975902fb50d596de [sys.catalog]: This master's current role is: LEADER
I20260926 20:48:46.828770 392 external_mini_cluster.cc:1428] Setting key 0c7aae053ddf0c41b666c759e704b21b
I20260926 20:48:46.828778 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55eec3ae1fe3486f975902fb50d596de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55eec3ae1fe3486f975902fb50d596de. Latest consensus state: current_term: 1 leader_uuid: "55eec3ae1fe3486f975902fb50d596de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55eec3ae1fe3486f975902fb50d596de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37663 } } }
I20260926 20:48:46.828835 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55eec3ae1fe3486f975902fb50d596de [sys.catalog]: This master's current role is: LEADER
I20260926 20:48:46.828963 491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:48:46.829434 491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:48:46.831115 491 catalog_manager.cc:1347] Generated new cluster ID: bff73ce3ac104418b950e4950c3560ac
I20260926 20:48:46.831166 491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:48:46.837668 491 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:48:46.838389 491 catalog_manager.cc:1504] Loading token signing keys...
Sep 26 20:48:46 dist-test-slave-hwlj krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:48:46.844667 491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55eec3ae1fe3486f975902fb50d596de: Generated new TSK 0
I20260926 20:48:46.844846 491 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:48:46.845680 502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:46.830065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53244 (local address 127.0.98.62:37663)
0926 20:48:46.830309 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:46.830317 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:46.830349 (+ 32us) server_negotiation.cc:407] Connection header received
0926 20:48:46.832403 (+ 2054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:46.832410 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:46.832467 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:46.832595 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:46.834612 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:46.836004 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:46.837580 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:46.838083 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:46.838214 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:46.841601 (+ 3387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:46.841605 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:46.841607 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:46.841622 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:46.843554 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:46.844127 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:46.844130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:46.844130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:46.844212 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:46.844529 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:46.844530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:46.844530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:46.844826 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:46.844962 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:46.845299 (+ 337us) server_negotiation.cc:299] Negotiation successful
0926 20:48:46.845450 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":204,"thread_start_us":72,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 20:48:46.871302 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37663
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:48:46.953256 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:48:46.953462 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:48:46.953488 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:48:46.954713 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:48:46.954759 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:48:46.954787 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:48:46.956385 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37663
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:48:46.956635 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:48:46.956854 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:48:46.962384 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:48:46.962523 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:48:46.962523 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:48:46.962589 506 server_base.cc:1034] running on GCE node
I20260926 20:48:46.962920 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:48:46.963176 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:48:46.964341 506 hybrid_clock.cc:648] HybridClock initialized: now 1790455726964342 us; error 43 us; skew 500 ppm
Sep 26 20:48:46 dist-test-slave-hwlj krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:48:46.966852 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 20:48:46.967319 506 webserver.cc:460] Webserver started at http://127.0.98.1:35669/ using document root <none> and password file <none>
I20260926 20:48:46.967506 506 fs_manager.cc:359] Metadata directory not provided
I20260926 20:48:46.967546 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:48:46.967635 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:48:46.968729 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/ts-0/data/instance:
uuid: "290047f82b854249b096a3005f1ef2bc"
format_stamp: "Formatted at 2026-09-26 20:48:46 on dist-test-slave-hwlj"
server_key: "57090caef7829c5d7cccecd797ed670d"
server_key_iv: "45a1d84695a1eaab8defe6a746ad7c59"
server_key_version: "encryptionkey@0"
I20260926 20:48:46.969451 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/ts-0/wal/instance:
uuid: "290047f82b854249b096a3005f1ef2bc"
format_stamp: "Formatted at 2026-09-26 20:48:46 on dist-test-slave-hwlj"
server_key: "57090caef7829c5d7cccecd797ed670d"
server_key_iv: "45a1d84695a1eaab8defe6a746ad7c59"
server_key_version: "encryptionkey@0"
I20260926 20:48:46.970918 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:48:46.971670 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:48:46.971896 506 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:48:46.971987 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/ts-0/wal
uuid: "290047f82b854249b096a3005f1ef2bc"
format_stamp: "Formatted at 2026-09-26 20:48:46 on dist-test-slave-hwlj"
server_key: "57090caef7829c5d7cccecd797ed670d"
server_key_iv: "45a1d84695a1eaab8defe6a746ad7c59"
server_key_version: "encryptionkey@0"
I20260926 20:48:46.972043 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:48:46.987247 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:48:46.989459 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:48:46.989574 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:48:46.989768 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:48:46.990085 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:48:46.990119 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:48:46.990144 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:48:46.990155 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:48:46.995879 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43247
I20260926 20:48:46.995965 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43247 every 8 connection(s)
I20260926 20:48:46.996306 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 20:48:46 dist-test-slave-hwlj krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:48:47.005735 502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:46.997122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55413 (local address 127.0.98.62:37663)
0926 20:48:46.997191 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:46.997193 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:46.997376 (+ 183us) server_negotiation.cc:407] Connection header received
0926 20:48:46.997876 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:46.997877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:46.997887 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:46.997950 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:46.998903 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:46.999414 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:47.000477 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:47.000701 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:47.000760 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:47.002841 (+ 2081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:47.002843 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:47.002843 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:47.002850 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:47.004414 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:47.004758 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:47.004759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:47.004759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:47.004809 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:47.005055 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:47.005056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:47.005056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:47.005251 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:47.005330 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:47.005599 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 20:48:47.005669 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 20:48:47.005913 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:46.997094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.1:55413)
0926 20:48:46.997245 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:46.997287 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:46.997795 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:46.998008 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:46.998012 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:46.998570 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:48:46.998774 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:46.998779 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:46.999506 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:46.999507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:47.000397 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:47.000400 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:47.000788 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:47.000789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:47.000893 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:47.001361 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:48:47.001369 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:48:47.002761 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:48:47.004480 (+ 1719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:47.004486 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:47.004488 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:47.004695 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:47.004864 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:47.004865 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:47.004865 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:47.004960 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:47.005327 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:48:47.005331 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:48:47.005442 (+ 111us) client_negotiation.cc:769] Sending connection context
0926 20:48:47.005567 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 20:48:47.005686 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":78,"threads_started":1}
I20260926 20:48:47.006316 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 506
I20260926 20:48:47.006415 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/ts-0/wal/instance
I20260926 20:48:47.006484 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37663
I20260926 20:48:47.006577 635 heartbeater.cc:464] Registering TS with master...
I20260926 20:48:47.006604 392 external_mini_cluster.cc:1428] Setting key 7d232684dda8b67756e6c6fdbdc74d27
I20260926 20:48:47.006775 635 heartbeater.cc:511] Master 127.0.98.62:37663 requested a full tablet report, sending...
I20260926 20:48:47.007337 443 ts_manager.cc:194] Registered new tserver with Master: 290047f82b854249b096a3005f1ef2bc (127.0.98.1:43247)
I20260926 20:48:47.008026 443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55413
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 20:48:47.031224 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37663
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:48:47.113135 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:48:47.113349 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:48:47.113373 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:48:47.114552 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:48:47.114595 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:48:47.114622 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:48:47.116154 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37663
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:48:47.116402 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:48:47.116652 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:48:47.121798 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:48:47.122115 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:48:47.122112 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:48:47.122298 640 server_base.cc:1034] running on GCE node
I20260926 20:48:47.122510 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:48:47.122757 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:48:47.123950 640 hybrid_clock.cc:648] HybridClock initialized: now 1790455727123932 us; error 29 us; skew 500 ppm
Sep 26 20:48:47 dist-test-slave-hwlj krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455727, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:48:47.126425 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 20:48:47.126869 640 webserver.cc:460] Webserver started at http://127.0.98.2:40795/ using document root <none> and password file <none>
I20260926 20:48:47.127089 640 fs_manager.cc:359] Metadata directory not provided
I20260926 20:48:47.127152 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:48:47.127298 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:48:47.128427 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/ts-1/data/instance:
uuid: "c902c29b96b5491f90b94fc0eea738dd"
format_stamp: "Formatted at 2026-09-26 20:48:47 on dist-test-slave-hwlj"
server_key: "99ba39cf23c00110f10174096d11a39e"
server_key_iv: "a38b2a9a5bab285d009bfb0b6cac6a21"
server_key_version: "encryptionkey@0"
I20260926 20:48:47.128827 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c902c29b96b5491f90b94fc0eea738dd"
format_stamp: "Formatted at 2026-09-26 20:48:47 on dist-test-slave-hwlj"
server_key: "99ba39cf23c00110f10174096d11a39e"
server_key_iv: "a38b2a9a5bab285d009bfb0b6cac6a21"
server_key_version: "encryptionkey@0"
I20260926 20:48:47.130196 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:48:47.130887 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:48:47.131057 640 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:48:47.131151 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/ts-1/wal
uuid: "c902c29b96b5491f90b94fc0eea738dd"
format_stamp: "Formatted at 2026-09-26 20:48:47 on dist-test-slave-hwlj"
server_key: "99ba39cf23c00110f10174096d11a39e"
server_key_iv: "a38b2a9a5bab285d009bfb0b6cac6a21"
server_key_version: "encryptionkey@0"
I20260926 20:48:47.131228 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:48:47.156322 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:48:47.158313 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:48:47.158478 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:48:47.158732 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:48:47.159050 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:48:47.159102 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:48:47.159145 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:48:47.159169 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:48:47.165268 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39389
I20260926 20:48:47.165347 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39389 every 8 connection(s)
I20260926 20:48:47.165657 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
I20260926 20:48:47.166041 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 640
I20260926 20:48:47.166143 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/ts-1/wal/instance
I20260926 20:48:47.166342 392 external_mini_cluster.cc:1428] Setting key b39013e509ea2b3adb2b5e23473b89b4
Sep 26 20:48:47 dist-test-slave-hwlj krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455727, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 20:48:47.176487 502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:47.166429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58431 (local address 127.0.98.62:37663)
0926 20:48:47.166477 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:47.166480 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:47.166706 (+ 226us) server_negotiation.cc:407] Connection header received
0926 20:48:47.167312 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:47.167313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:47.167325 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:47.167369 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:47.168479 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:47.169038 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:47.170667 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:47.170863 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:47.170920 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:47.173453 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:47.173455 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:47.173455 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:47.173463 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:47.175067 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:47.175476 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:47.175478 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:47.175478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:47.175540 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:47.175821 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:47.175822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:47.175822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:47.175945 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:47.176025 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:47.176299 (+ 274us) server_negotiation.cc:299] Negotiation successful
0926 20:48:47.176391 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 20:48:47.176676 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:47.166405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.2:58431)
0926 20:48:47.166563 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:47.166605 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:47.167203 (+ 598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:47.167440 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:47.167445 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:47.168141 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:48:47.168346 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:47.168351 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:47.169175 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:47.169176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:47.170561 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:47.170564 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:47.170952 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:47.170953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:47.171109 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:47.171658 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:48:47.171669 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:48:47.173354 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:48:47.175154 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:47.175160 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:47.175163 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:47.175385 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:47.175607 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:47.175608 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:47.175608 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:47.175731 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:47.176014 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:48:47.176021 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:48:47.176166 (+ 145us) client_negotiation.cc:769] Sending connection context
0926 20:48:47.176278 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 20:48:47.176440 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
I20260926 20:48:47.177151 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37663
I20260926 20:48:47.177248 769 heartbeater.cc:464] Registering TS with master...
I20260926 20:48:47.177397 769 heartbeater.cc:511] Master 127.0.98.62:37663 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 20:48:47.177767 443 ts_manager.cc:194] Registered new tserver with Master: c902c29b96b5491f90b94fc0eea738dd (127.0.98.2:39389)
I20260926 20:48:47.178223 443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58431
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 20:48:47.194507 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37663
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:48:47.277777 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:48:47.278115 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:48:47.278195 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:48:47.279858 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:48:47.279950 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:48:47.280027 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:48:47.282128 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37663
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:48:47.282390 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:48:47.282652 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:48:47.288909 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:48:47.288911 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:48:47.289119 774 server_base.cc:1034] running on GCE node
W20260926 20:48:47.288911 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:48:47.289433 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:48:47.289750 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:48:47.290902 774 hybrid_clock.cc:648] HybridClock initialized: now 1790455727290887 us; error 31 us; skew 500 ppm
Sep 26 20:48:47 dist-test-slave-hwlj krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455727, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:48:47.293565 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 20:48:47.294040 774 webserver.cc:460] Webserver started at http://127.0.98.3:39783/ using document root <none> and password file <none>
I20260926 20:48:47.294302 774 fs_manager.cc:359] Metadata directory not provided
I20260926 20:48:47.294371 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:48:47.294516 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:48:47.295569 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/ts-2/data/instance:
uuid: "7abaeeeaa2c6446b928529f28eb65d21"
format_stamp: "Formatted at 2026-09-26 20:48:47 on dist-test-slave-hwlj"
server_key: "99491b914d376bc6c1bee4d25f60f5c4"
server_key_iv: "846d76bda352ed24019dc12cc26a3389"
server_key_version: "encryptionkey@0"
I20260926 20:48:47.295953 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7abaeeeaa2c6446b928529f28eb65d21"
format_stamp: "Formatted at 2026-09-26 20:48:47 on dist-test-slave-hwlj"
server_key: "99491b914d376bc6c1bee4d25f60f5c4"
server_key_iv: "846d76bda352ed24019dc12cc26a3389"
server_key_version: "encryptionkey@0"
I20260926 20:48:47.297291 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:48:47.298030 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:48:47.298188 774 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 20:48:47.298290 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/ts-2/wal
uuid: "7abaeeeaa2c6446b928529f28eb65d21"
format_stamp: "Formatted at 2026-09-26 20:48:47 on dist-test-slave-hwlj"
server_key: "99491b914d376bc6c1bee4d25f60f5c4"
server_key_iv: "846d76bda352ed24019dc12cc26a3389"
server_key_version: "encryptionkey@0"
I20260926 20:48:47.298362 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:48:47.323015 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:48:47.324985 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:48:47.325150 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:48:47.325398 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:48:47.325703 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:48:47.325757 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:48:47.325800 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:48:47.325826 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:48:47.333005 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39551
I20260926 20:48:47.333046 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39551 every 8 connection(s)
I20260926 20:48:47.333369 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 20:48:47 dist-test-slave-hwlj krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455727, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:48:47.341738 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 774
I20260926 20:48:47.341830 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455725523895-392-0/minicluster-data/ts-2/wal/instance
I20260926 20:48:47.342029 392 external_mini_cluster.cc:1428] Setting key b36331bb671d41eceb94cef8754adfee
I20260926 20:48:47.342519 502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:47.334082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48151 (local address 127.0.98.62:37663)
0926 20:48:47.334174 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:47.334176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:47.334349 (+ 173us) server_negotiation.cc:407] Connection header received
0926 20:48:47.334799 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:47.334800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:47.334809 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:47.334907 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:47.335884 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:47.336367 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:47.337324 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:47.337498 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:47.337547 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:47.339647 (+ 2100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:47.339648 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:47.339649 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:47.339655 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:47.340945 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:47.341360 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:47.341361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:47.341361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:47.341409 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:47.341706 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:47.341707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:47.341707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:47.341855 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:47.341945 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:47.342335 (+ 390us) server_negotiation.cc:299] Negotiation successful
0926 20:48:47.342416 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 20:48:47.342700 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:47.334049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.3:48151)
0926 20:48:47.334228 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:47.334264 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:47.334722 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:47.334984 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:47.334987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:47.335544 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:48:47.335795 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:47.335803 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:47.336444 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:47.336445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:47.337247 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:47.337249 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:47.337567 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:47.337567 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:47.337665 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:47.338143 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:48:47.338151 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:48:47.339548 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:48:47.341036 (+ 1488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:47.341042 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:47.341043 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:47.341291 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:47.341466 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:47.341466 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:47.341467 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:47.341618 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:47.341966 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:48:47.341972 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:48:47.342171 (+ 199us) client_negotiation.cc:769] Sending connection context
0926 20:48:47.342300 (+ 129us) client_negotiation.cc:240] Negotiation successful
0926 20:48:47.342421 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":93,"threads_started":1}
I20260926 20:48:47.343173 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37663
I20260926 20:48:47.343267 903 heartbeater.cc:464] Registering TS with master...
I20260926 20:48:47.343485 903 heartbeater.cc:511] Master 127.0.98.62:37663 requested a full tablet report, sending...
I20260926 20:48:47.343861 443 ts_manager.cc:194] Registered new tserver with Master: 7abaeeeaa2c6446b928529f28eb65d21 (127.0.98.3:39551)
I20260926 20:48:47.344241 443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48151
I20260926 20:48:47.354125 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 20:48:47 dist-test-slave-hwlj krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455727, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 20:48:47 dist-test-slave-hwlj krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455727, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 20:48:47.377501 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:47.369125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59656 (local address 127.0.98.1:43247)
0926 20:48:47.369661 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:47.369666 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:47.369693 (+ 27us) server_negotiation.cc:407] Connection header received
0926 20:48:47.369729 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:47.369731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:47.369744 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:47.369825 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:47.370392 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:47.371135 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:47.371949 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:47.372267 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:47.372350 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:47.374846 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:47.374853 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:47.374853 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:47.374860 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:47.376448 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:47.376767 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:47.376769 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:47.376770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:47.376837 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:47.377002 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:47.377002 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:47.377003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:47.377147 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:47.377251 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:47.377320 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 20:48:47.377380 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":66,"threads_started":1}
W20260926 20:48:47.377866 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59656
Sep 26 20:48:47 dist-test-slave-hwlj krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 20:48:47 dist-test-slave-hwlj krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 20:48:47.399272 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:47.392640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59672 (local address 127.0.98.1:43247)
0926 20:48:47.392702 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:47.392704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:47.392779 (+ 75us) server_negotiation.cc:407] Connection header received
0926 20:48:47.392853 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:47.392854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:47.392865 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:47.392914 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:47.393467 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:47.393948 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:47.394666 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:47.394861 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:47.394913 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:47.397154 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:47.397156 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:47.397156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:47.397164 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:47.398355 (+ 1191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:47.398641 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:47.398642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:47.398642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:47.398688 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:47.398841 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:47.398841 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:47.398842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:47.398927 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:47.399010 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:47.399074 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 20:48:47.399154 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 20:48:47.399921 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 506
I20260926 20:48:47.404990 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 640
I20260926 20:48:47.410939 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 774
I20260926 20:48:47.414901 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 412
2026-09-26T20:48:47Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1898 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:48:47 dist-test-slave-hwlj krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:48:47 dist-test-slave-hwlj krb5kdc[917](info): set up 2 sockets
Sep 26 20:48:47 dist-test-slave-hwlj krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260926 20:48:48.471239 392 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/krb5kdc/test-user.keytab.
Sep 26 20:48:48 dist-test-slave-hwlj krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:48:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:48:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 20:48:48.588467 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40085
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40085
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:48:48.668491 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:48:48.668691 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:48:48.668713 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:48:48.669778 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:48:48.669811 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:48:48.669826 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:48:48.669838 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:48:48.669849 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:48:48.671238 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:48:48.671422 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:48:48.671588 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:48:48.676429 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:48:48.676443 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:48:48.676483 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:48:48.676586 930 server_base.cc:1034] running on GCE node
I20260926 20:48:48.676847 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:48:48.677139 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:48:48.678308 930 hybrid_clock.cc:648] HybridClock initialized: now 1790455728678291 us; error 27 us; skew 500 ppm
Sep 26 20:48:48 dist-test-slave-hwlj krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:48:48.680840 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:48:48.681308 930 webserver.cc:460] Webserver started at http://127.0.98.62:44799/ using document root <none> and password file <none>
I20260926 20:48:48.681556 930 fs_manager.cc:359] Metadata directory not provided
I20260926 20:48:48.681627 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:48:48.681777 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:48:48.682834 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/master-0/data/instance:
uuid: "e9540f620a264e6ba4062454f8cc4339"
format_stamp: "Formatted at 2026-09-26 20:48:48 on dist-test-slave-hwlj"
server_key: "1046b044e71c1be81cdcadd0daa14d0c"
server_key_iv: "3fbd4ccf2226a704cb8a19e7a6e53288"
server_key_version: "encryptionkey@0"
I20260926 20:48:48.683199 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/master-0/wal/instance:
uuid: "e9540f620a264e6ba4062454f8cc4339"
format_stamp: "Formatted at 2026-09-26 20:48:48 on dist-test-slave-hwlj"
server_key: "1046b044e71c1be81cdcadd0daa14d0c"
server_key_iv: "3fbd4ccf2226a704cb8a19e7a6e53288"
server_key_version: "encryptionkey@0"
I20260926 20:48:48.684644 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:48:48.685403 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:48:48.685731 930 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:48:48.685844 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "e9540f620a264e6ba4062454f8cc4339"
format_stamp: "Formatted at 2026-09-26 20:48:48 on dist-test-slave-hwlj"
server_key: "1046b044e71c1be81cdcadd0daa14d0c"
server_key_iv: "3fbd4ccf2226a704cb8a19e7a6e53288"
server_key_version: "encryptionkey@0"
I20260926 20:48:48.685920 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:48:48.693553 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:48:48.695329 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:48:48.695487 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:48:48.699369 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40085
I20260926 20:48:48.699440 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40085 every 8 connection(s)
I20260926 20:48:48.699858 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:48:48.700701 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:48:48.702943 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9540f620a264e6ba4062454f8cc4339: Bootstrap starting.
I20260926 20:48:48.702955 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 930
I20260926 20:48:48.703038 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/master-0/wal/instance
I20260926 20:48:48.703181 392 external_mini_cluster.cc:1428] Setting key 3a6c9a6ecd3631c236f687faf08b6726
I20260926 20:48:48.703670 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9540f620a264e6ba4062454f8cc4339: Neither blocks nor log segments found. Creating new log.
I20260926 20:48:48.704020 998 log.cc:824] T 00000000000000000000000000000000 P e9540f620a264e6ba4062454f8cc4339: Log is configured to *not* fsync() on all Append() calls
I20260926 20:48:48.704753 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9540f620a264e6ba4062454f8cc4339: No bootstrap required, opened a new log
I20260926 20:48:48.706911 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9540f620a264e6ba4062454f8cc4339 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9540f620a264e6ba4062454f8cc4339" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40085 } }
I20260926 20:48:48.707082 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9540f620a264e6ba4062454f8cc4339 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:48:48.707113 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9540f620a264e6ba4062454f8cc4339 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9540f620a264e6ba4062454f8cc4339, State: Initialized, Role: FOLLOWER
I20260926 20:48:48.707234 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9540f620a264e6ba4062454f8cc4339 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9540f620a264e6ba4062454f8cc4339" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40085 } }
I20260926 20:48:48.707302 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9540f620a264e6ba4062454f8cc4339 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:48:48.707329 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9540f620a264e6ba4062454f8cc4339 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:48:48.707371 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9540f620a264e6ba4062454f8cc4339 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:48:48.708012 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9540f620a264e6ba4062454f8cc4339 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9540f620a264e6ba4062454f8cc4339" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40085 } }
I20260926 20:48:48.708101 998 leader_election.cc:302] T 00000000000000000000000000000000 P e9540f620a264e6ba4062454f8cc4339 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9540f620a264e6ba4062454f8cc4339; no voters:
I20260926 20:48:48.708312 998 leader_election.cc:288] T 00000000000000000000000000000000 P e9540f620a264e6ba4062454f8cc4339 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:48:48.708341 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9540f620a264e6ba4062454f8cc4339 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:48:48.708449 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9540f620a264e6ba4062454f8cc4339 [term 1 LEADER]: Becoming Leader. State: Replica: e9540f620a264e6ba4062454f8cc4339, State: Running, Role: LEADER
I20260926 20:48:48.708526 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9540f620a264e6ba4062454f8cc4339 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9540f620a264e6ba4062454f8cc4339" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40085 } }
I20260926 20:48:48.708882 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9540f620a264e6ba4062454f8cc4339 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:48:48.708860 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9540f620a264e6ba4062454f8cc4339 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9540f620a264e6ba4062454f8cc4339. Latest consensus state: current_term: 1 leader_uuid: "e9540f620a264e6ba4062454f8cc4339" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9540f620a264e6ba4062454f8cc4339" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40085 } } }
I20260926 20:48:48.708992 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9540f620a264e6ba4062454f8cc4339 [sys.catalog]: This master's current role is: LEADER
I20260926 20:48:48.708873 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9540f620a264e6ba4062454f8cc4339 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9540f620a264e6ba4062454f8cc4339" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9540f620a264e6ba4062454f8cc4339" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40085 } } }
I20260926 20:48:48.709210 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9540f620a264e6ba4062454f8cc4339 [sys.catalog]: This master's current role is: LEADER
I20260926 20:48:48.709595 1007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:48:48.710119 1007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 20:48:48 dist-test-slave-hwlj krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:48:48.712028 1007 catalog_manager.cc:1347] Generated new cluster ID: 96f50f27c8aa42d9b4f6baaebc9a84a0
I20260926 20:48:48.712085 1007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:48:48.715240 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:48.703739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45782 (local address 127.0.98.62:40085)
0926 20:48:48.703925 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:48.703934 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:48.703973 (+ 39us) server_negotiation.cc:407] Connection header received
0926 20:48:48.704452 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:48.704460 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:48.704509 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:48.704646 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:48.705421 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:48.706491 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:48.707566 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:48.708111 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:48.708240 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:48.711386 (+ 3146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:48.711390 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:48.711390 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:48.711403 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:48.713526 (+ 2123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:48.713947 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:48.713950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:48.713950 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:48.714071 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:48.714303 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:48.714304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:48.714304 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:48.714612 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:48.714766 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:48.714872 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 20:48:48.715011 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":85,"threads_started":1}
I20260926 20:48:48.722968 1007 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:48:48.723498 1007 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:48:48.725770 1007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9540f620a264e6ba4062454f8cc4339: Generated new TSK 0
I20260926 20:48:48.725970 1007 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 20:48:48.754959 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40085
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:48:48.847982 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:48:48.848253 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:48:48.848284 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:48:48.849632 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:48:48.849673 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:48:48.849701 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:48:48.851164 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40085
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:48:48.851362 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:48:48.851594 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:48:48.856513 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:48:48.856655 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:48:48.856508 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:48:48.856735 1024 server_base.cc:1034] running on GCE node
I20260926 20:48:48.856923 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:48:48.857120 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:48:48.858265 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790455728858236 us; error 45 us; skew 500 ppm
Sep 26 20:48:48 dist-test-slave-hwlj krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:48:48.861294 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 20:48:48.861770 1024 webserver.cc:460] Webserver started at http://127.0.98.1:37591/ using document root <none> and password file <none>
I20260926 20:48:48.862003 1024 fs_manager.cc:359] Metadata directory not provided
I20260926 20:48:48.862056 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:48:48.862149 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:48:48.863157 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/ts-0/data/instance:
uuid: "c4228f7905ab45528b5739a1a69f9c7d"
format_stamp: "Formatted at 2026-09-26 20:48:48 on dist-test-slave-hwlj"
server_key: "01b72e202575f4883b9a7b8401abec46"
server_key_iv: "9514716b84fd2d0dd5afebe182e031d7"
server_key_version: "encryptionkey@0"
I20260926 20:48:48.863492 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c4228f7905ab45528b5739a1a69f9c7d"
format_stamp: "Formatted at 2026-09-26 20:48:48 on dist-test-slave-hwlj"
server_key: "01b72e202575f4883b9a7b8401abec46"
server_key_iv: "9514716b84fd2d0dd5afebe182e031d7"
server_key_version: "encryptionkey@0"
I20260926 20:48:48.864949 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:48:48.865669 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:48:48.865839 1024 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 20:48:48.865919 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/ts-0/wal
uuid: "c4228f7905ab45528b5739a1a69f9c7d"
format_stamp: "Formatted at 2026-09-26 20:48:48 on dist-test-slave-hwlj"
server_key: "01b72e202575f4883b9a7b8401abec46"
server_key_iv: "9514716b84fd2d0dd5afebe182e031d7"
server_key_version: "encryptionkey@0"
I20260926 20:48:48.865974 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:48:48.877405 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:48:48.879739 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:48:48.879976 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:48:48.880223 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:48:48.880532 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:48:48.880565 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:48:48.880589 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:48:48.880601 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:48:48.886554 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44609
I20260926 20:48:48.886647 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44609 every 8 connection(s)
I20260926 20:48:48.886951 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:48:48.890314 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 1024
I20260926 20:48:48.890424 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/ts-0/wal/instance
I20260926 20:48:48.890600 392 external_mini_cluster.cc:1428] Setting key 2b9d040a0f5fdea211b051ae2b81c66c
Sep 26 20:48:48 dist-test-slave-hwlj krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:48:48.898047 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:48.887658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38127 (local address 127.0.98.62:40085)
0926 20:48:48.887729 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:48.887732 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:48.887921 (+ 189us) server_negotiation.cc:407] Connection header received
0926 20:48:48.888470 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:48.888471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:48.888480 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:48.888531 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:48.889510 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:48.890098 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:48.891847 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:48.892048 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:48.892111 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:48.895081 (+ 2970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:48.895083 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:48.895083 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:48.895091 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:48.896635 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:48.897029 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:48.897030 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:48.897030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:48.897091 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:48.897358 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:48.897359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:48.897359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:48.897522 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:48.897595 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:48.897869 (+ 274us) server_negotiation.cc:299] Negotiation successful
0926 20:48:48.897956 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 20:48:48.898252 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:48.887608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40085 (local address 127.0.98.1:38127)
0926 20:48:48.887800 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:48.887835 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:48.888360 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:48.888593 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:48.888597 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:48.889235 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:48:48.889425 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:48.889430 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:48.890260 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:48.890262 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:48.891741 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:48.891745 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:48.892135 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:48.892136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:48.892301 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:48.892883 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:48:48.892892 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:48:48.894983 (+ 2091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:48:48.896729 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:48.896735 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:48.896737 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:48.896955 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:48.897155 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:48.897156 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:48.897156 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:48.897278 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:48.897584 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:48:48.897590 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:48:48.897739 (+ 149us) client_negotiation.cc:769] Sending connection context
0926 20:48:48.897847 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 20:48:48.898008 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":90,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 20:48:48.898720 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40085
I20260926 20:48:48.898825 1153 heartbeater.cc:464] Registering TS with master...
I20260926 20:48:48.899004 1153 heartbeater.cc:511] Master 127.0.98.62:40085 requested a full tablet report, sending...
I20260926 20:48:48.899645 962 ts_manager.cc:194] Registered new tserver with Master: c4228f7905ab45528b5739a1a69f9c7d (127.0.98.1:44609)
I20260926 20:48:48.900439 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38127
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
W20260926 20:48:49.636799 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.2: real 0.745s user 0.000s sys 0.003s
I20260926 20:48:49.637151 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40085
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:48:49.722662 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:48:49.722874 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:48:49.722900 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:48:49.724524 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:48:49.724579 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:48:49.724610 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:48:49.726143 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40085
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:48:49.726342 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:48:49.726531 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:48:49.731788 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:48:49.731773 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:48:49.731838 1158 server_base.cc:1034] running on GCE node
W20260926 20:48:49.731776 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:48:49.732178 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:48:49.732446 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:48:49.733639 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790455729733619 us; error 38 us; skew 500 ppm
Sep 26 20:48:49 dist-test-slave-hwlj krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455729, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:48:49.736361 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 20:48:49.736846 1158 webserver.cc:460] Webserver started at http://127.0.98.2:45157/ using document root <none> and password file <none>
I20260926 20:48:49.737098 1158 fs_manager.cc:359] Metadata directory not provided
I20260926 20:48:49.737146 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:48:49.737241 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:48:49.738281 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/ts-1/data/instance:
uuid: "cdc41b9889334d76a7018a9cac0e858f"
format_stamp: "Formatted at 2026-09-26 20:48:49 on dist-test-slave-hwlj"
server_key: "b03b876b02290dc0689e4fcdf22370a7"
server_key_iv: "a44ee5ee632b96a7725cb0a61e7e42e0"
server_key_version: "encryptionkey@0"
I20260926 20:48:49.738601 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/ts-1/wal/instance:
uuid: "cdc41b9889334d76a7018a9cac0e858f"
format_stamp: "Formatted at 2026-09-26 20:48:49 on dist-test-slave-hwlj"
server_key: "b03b876b02290dc0689e4fcdf22370a7"
server_key_iv: "a44ee5ee632b96a7725cb0a61e7e42e0"
server_key_version: "encryptionkey@0"
I20260926 20:48:49.740257 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:48:49.741127 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:48:49.741379 1158 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:48:49.741509 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/ts-1/wal
uuid: "cdc41b9889334d76a7018a9cac0e858f"
format_stamp: "Formatted at 2026-09-26 20:48:49 on dist-test-slave-hwlj"
server_key: "b03b876b02290dc0689e4fcdf22370a7"
server_key_iv: "a44ee5ee632b96a7725cb0a61e7e42e0"
server_key_version: "encryptionkey@0"
I20260926 20:48:49.741571 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:48:49.758317 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:48:49.761018 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:48:49.761185 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:48:49.761406 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:48:49.761823 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:48:49.761869 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:48:49.761929 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:48:49.761957 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:48:49.767966 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39433
I20260926 20:48:49.768054 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39433 every 8 connection(s)
I20260926 20:48:49.768501 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
I20260926 20:48:49.772233 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 1158
I20260926 20:48:49.772351 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/ts-1/wal/instance
I20260926 20:48:49.772527 392 external_mini_cluster.cc:1428] Setting key 9a11ad41280327ea42b465e7d8095a8d
Sep 26 20:48:49 dist-test-slave-hwlj krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455729, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:48:49.780217 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:49.769426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37221 (local address 127.0.98.62:40085)
0926 20:48:49.769549 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:49.769551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:49.769641 (+ 90us) server_negotiation.cc:407] Connection header received
0926 20:48:49.770172 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:49.770174 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:49.770186 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:49.770257 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:49.771228 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:49.771732 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:49.773091 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:49.773372 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:49.773433 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:49.776214 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:49.776216 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:49.776217 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:49.776226 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:49.778255 (+ 2029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:49.778749 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:49.778752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:49.778752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:49.778825 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:49.779275 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:49.779277 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:49.779277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:49.779446 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:49.779529 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:49.779978 (+ 449us) server_negotiation.cc:299] Negotiation successful
0926 20:48:49.780078 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":61,"threads_started":1}
I20260926 20:48:49.780292 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:49.769322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40085 (local address 127.0.98.2:37221)
0926 20:48:49.769484 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:49.769552 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:49.770072 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:49.770309 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:49.770313 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:49.770903 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:48:49.771102 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:49.771107 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:49.771936 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:49.771936 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:49.773007 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:49.773010 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:49.773469 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:49.773470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:49.773628 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:49.774171 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:48:49.774182 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:48:49.776082 (+ 1900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:48:49.778365 (+ 2283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:49.778371 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:49.778374 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:49.778656 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:49.779011 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:49.779012 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:49.779012 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:49.779184 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:49.779556 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:48:49.779562 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:48:49.779812 (+ 250us) client_negotiation.cc:769] Sending connection context
0926 20:48:49.779931 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 20:48:49.780070 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":84,"threads_started":1}
I20260926 20:48:49.780860 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40085
I20260926 20:48:49.780982 1287 heartbeater.cc:464] Registering TS with master...
I20260926 20:48:49.781214 1287 heartbeater.cc:511] Master 127.0.98.62:40085 requested a full tablet report, sending...
I20260926 20:48:49.781724 962 ts_manager.cc:194] Registered new tserver with Master: cdc41b9889334d76a7018a9cac0e858f (127.0.98.2:39433)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 20:48:49.782232 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37221
I20260926 20:48:49.901373 1153 heartbeater.cc:503] Master 127.0.98.62:40085 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 20:48:50.783126 1287 heartbeater.cc:503] Master 127.0.98.62:40085 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
W20260926 20:48:50.936471 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.3: real 1.163s user 0.000s sys 0.003s
I20260926 20:48:50.936894 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40085
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:48:51.018652 1293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:48:51.018851 1293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:48:51.018873 1293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:48:51.020077 1293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:48:51.020119 1293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:48:51.020186 1293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:48:51.021649 1293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40085
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:48:51.021889 1293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:48:51.022110 1293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:48:51.027159 1298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:48:51.027173 1301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:48:51.027173 1299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:48:51.027567 1293 server_base.cc:1034] running on GCE node
I20260926 20:48:51.027832 1293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:48:51.028090 1293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:48:51.029285 1293 hybrid_clock.cc:648] HybridClock initialized: now 1790455731029266 us; error 31 us; skew 500 ppm
Sep 26 20:48:51 dist-test-slave-hwlj krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455731, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:48:51.038650 1293 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 20:48:51.039208 1293 webserver.cc:460] Webserver started at http://127.0.98.3:39115/ using document root <none> and password file <none>
I20260926 20:48:51.039448 1293 fs_manager.cc:359] Metadata directory not provided
I20260926 20:48:51.039527 1293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:48:51.039680 1293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:48:51.040845 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/ts-2/data/instance:
uuid: "8cdc525fcd31402f8d9dac64113ae636"
format_stamp: "Formatted at 2026-09-26 20:48:51 on dist-test-slave-hwlj"
server_key: "5fe3985ffd021444924719bde45e60b2"
server_key_iv: "729933bb1f20f18a7fdfe5d141385b62"
server_key_version: "encryptionkey@0"
I20260926 20:48:51.041222 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8cdc525fcd31402f8d9dac64113ae636"
format_stamp: "Formatted at 2026-09-26 20:48:51 on dist-test-slave-hwlj"
server_key: "5fe3985ffd021444924719bde45e60b2"
server_key_iv: "729933bb1f20f18a7fdfe5d141385b62"
server_key_version: "encryptionkey@0"
I20260926 20:48:51.042699 1293 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:48:51.043529 1308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:48:51.043807 1293 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:48:51.043926 1293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/ts-2/wal
uuid: "8cdc525fcd31402f8d9dac64113ae636"
format_stamp: "Formatted at 2026-09-26 20:48:51 on dist-test-slave-hwlj"
server_key: "5fe3985ffd021444924719bde45e60b2"
server_key_iv: "729933bb1f20f18a7fdfe5d141385b62"
server_key_version: "encryptionkey@0"
I20260926 20:48:51.044019 1293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:48:51.063275 1293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:48:51.065732 1293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:48:51.065939 1293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:48:51.066243 1293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:48:51.066573 1293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:48:51.066632 1293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:48:51.066679 1293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:48:51.066705 1293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:48:51.073200 1293 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42395
I20260926 20:48:51.073240 1421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42395 every 8 connection(s)
I20260926 20:48:51.073653 1293 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 20:48:51 dist-test-slave-hwlj krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455731, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:48:51.083122 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 1293
I20260926 20:48:51.083225 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455725523895-392-0/minicluster-data/ts-2/wal/instance
I20260926 20:48:51.083442 392 external_mini_cluster.cc:1428] Setting key 75c9b275d7283e6eb86d3397ce744a98
I20260926 20:48:51.083873 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:51.074552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57289 (local address 127.0.98.62:40085)
0926 20:48:51.074729 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:51.074731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:51.074830 (+ 99us) server_negotiation.cc:407] Connection header received
0926 20:48:51.075347 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:51.075348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:51.075360 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:51.075438 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:51.076588 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:51.077114 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:51.078108 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:51.078307 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:51.078364 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:51.080662 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:51.080663 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:51.080663 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:51.080670 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:51.082264 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:51.082689 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:51.082690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:51.082690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:51.082747 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:51.083069 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:51.083071 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:51.083071 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:51.083244 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:51.083327 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:51.083628 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 20:48:51.083720 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":114,"threads_started":1}
I20260926 20:48:51.084132 1425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:51.074512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40085 (local address 127.0.98.3:57289)
0926 20:48:51.074686 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:51.074736 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:51.075223 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:51.075510 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:51.075514 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:51.076252 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:48:51.076458 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:51.076463 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:51.077205 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:51.077206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:51.078014 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:51.078016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:51.078385 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:51.078386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:51.078484 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:51.078991 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:48:51.078999 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:48:51.080568 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:48:51.082366 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:51.082372 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:51.082374 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:51.082609 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:51.082824 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:51.082825 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:51.082825 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:51.082938 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:51.083330 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:48:51.083337 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:48:51.083471 (+ 134us) client_negotiation.cc:769] Sending connection context
0926 20:48:51.083601 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 20:48:51.083814 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":100,"threads_started":1}
I20260926 20:48:51.084563 1422 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40085
I20260926 20:48:51.084789 1422 heartbeater.cc:464] Registering TS with master...
I20260926 20:48:51.085026 1422 heartbeater.cc:511] Master 127.0.98.62:40085 requested a full tablet report, sending...
I20260926 20:48:51.085474 961 ts_manager.cc:194] Registered new tserver with Master: 8cdc525fcd31402f8d9dac64113ae636 (127.0.98.3:42395)
I20260926 20:48:51.085860 961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57289
I20260926 20:48:51.095860 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:48:51.104475 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:51.097726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60722 (local address 127.0.98.62:40085)
0926 20:48:51.097804 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:51.097806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:51.097815 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:48:51.097858 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:51.097859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:51.097869 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:51.097931 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:51.098629 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:51.099210 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:51.100033 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:51.100236 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:51.100300 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:51.101772 (+ 1472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:51.101773 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:51.101773 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:51.101781 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:51.103269 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:51.103593 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:51.103593 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:51.103594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:51.103642 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:51.103990 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:51.103990 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:51.103991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:51.104154 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:51.104243 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:51.104339 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 20:48:51.104405 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 20:48:51.107064 961 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60722:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:48:51.108712 961 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:48:51.115535 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:51.111410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44609 (local address 127.0.0.1:44042)
0926 20:48:51.112030 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:51.112039 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:51.112118 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:51.112359 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:51.112359 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:51.112361 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:48:51.112395 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:51.112396 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:51.113218 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:51.113219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:51.114407 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:51.114410 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:51.115238 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:51.115239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:51.115295 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:51.115305 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 20:48:51.115392 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 20:48:51.115443 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":93,"threads_started":1}
I20260926 20:48:51.115689 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:51.111404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42395 (local address 127.0.0.1:47358)
0926 20:48:51.111685 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:51.111724 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:51.111851 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:51.112118 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:51.112121 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:51.112124 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:48:51.112199 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:51.112201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:51.113659 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:51.113660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:51.114496 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:51.114498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:51.115391 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:51.115392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:51.115446 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:51.115448 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 20:48:51.115502 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 20:48:51.115533 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":92,"threads_started":1}
I20260926 20:48:51.115913 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:51.111687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44042 (local address 127.0.98.1:44609)
0926 20:48:51.111842 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:51.111846 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:51.112088 (+ 242us) server_negotiation.cc:407] Connection header received
0926 20:48:51.112201 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:51.112203 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:51.112220 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:51.112271 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:48:51.112446 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:51.113109 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:51.114510 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:51.115119 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:51.115155 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:51.115193 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:51.115769 (+ 576us) server_negotiation.cc:299] Negotiation successful
0926 20:48:51.115821 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":77,"threads_started":1}
I20260926 20:48:51.115957 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:51.111667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47358 (local address 127.0.98.3:42395)
0926 20:48:51.111927 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:51.111931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:51.111941 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:48:51.111977 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:51.111979 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:51.112003 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:51.112112 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:48:51.112257 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:51.113572 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:51.114577 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:51.115291 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:51.115373 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:51.115425 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:51.115513 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 20:48:51.115554 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":82,"threads_started":1}
I20260926 20:48:51.115955 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:51.111502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47464 (local address 127.0.98.2:39433)
0926 20:48:51.111640 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:51.111646 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:51.111948 (+ 302us) server_negotiation.cc:407] Connection header received
0926 20:48:51.112024 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:51.112027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:51.112045 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:51.112161 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:48:51.112397 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:51.113507 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:51.114793 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:51.115475 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:51.115538 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:51.115596 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:51.115840 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 20:48:51.115880 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
I20260926 20:48:51.116201 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:51.111407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39433 (local address 127.0.0.1:47464)
0926 20:48:51.111846 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:51.111856 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:51.111948 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:51.112229 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:51.112231 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:51.112233 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:48:51.112282 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:51.112283 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:51.113618 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:51.113619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:51.114701 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:51.114703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:51.115634 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:51.115635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:51.115712 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:51.115715 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:48:51.116084 (+ 369us) client_negotiation.cc:240] Negotiation successful
0926 20:48:51.116134 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":95,"threads_started":1}
I20260926 20:48:51.116696 1087 tablet_service.cc:1467] Processing CreateTablet for tablet c0356eb5fa434477874da77246262c9d (DEFAULT_TABLE table=test-table [id=c60913188c834cfab848f8b7c83400ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:48:51.116941 1221 tablet_service.cc:1467] Processing CreateTablet for tablet c0356eb5fa434477874da77246262c9d (DEFAULT_TABLE table=test-table [id=c60913188c834cfab848f8b7c83400ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:48:51.117089 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0356eb5fa434477874da77246262c9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:48:51.117285 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0356eb5fa434477874da77246262c9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:48:51.119618 1442 tablet_bootstrap.cc:492] T c0356eb5fa434477874da77246262c9d P c4228f7905ab45528b5739a1a69f9c7d: Bootstrap starting.
I20260926 20:48:51.120055 1356 tablet_service.cc:1467] Processing CreateTablet for tablet c0356eb5fa434477874da77246262c9d (DEFAULT_TABLE table=test-table [id=c60913188c834cfab848f8b7c83400ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:48:51.120363 1356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0356eb5fa434477874da77246262c9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:48:51.120381 1442 tablet_bootstrap.cc:654] T c0356eb5fa434477874da77246262c9d P c4228f7905ab45528b5739a1a69f9c7d: Neither blocks nor log segments found. Creating new log.
I20260926 20:48:51.120697 1442 log.cc:824] T c0356eb5fa434477874da77246262c9d P c4228f7905ab45528b5739a1a69f9c7d: Log is configured to *not* fsync() on all Append() calls
I20260926 20:48:51.121032 1441 tablet_bootstrap.cc:492] T c0356eb5fa434477874da77246262c9d P cdc41b9889334d76a7018a9cac0e858f: Bootstrap starting.
I20260926 20:48:51.121397 1442 tablet_bootstrap.cc:492] T c0356eb5fa434477874da77246262c9d P c4228f7905ab45528b5739a1a69f9c7d: No bootstrap required, opened a new log
I20260926 20:48:51.121448 1442 ts_tablet_manager.cc:1397] T c0356eb5fa434477874da77246262c9d P c4228f7905ab45528b5739a1a69f9c7d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:48:51.122838 1444 tablet_bootstrap.cc:492] T c0356eb5fa434477874da77246262c9d P 8cdc525fcd31402f8d9dac64113ae636: Bootstrap starting.
I20260926 20:48:51.123526 1444 tablet_bootstrap.cc:654] T c0356eb5fa434477874da77246262c9d P 8cdc525fcd31402f8d9dac64113ae636: Neither blocks nor log segments found. Creating new log.
I20260926 20:48:51.123806 1442 raft_consensus.cc:348] T c0356eb5fa434477874da77246262c9d P c4228f7905ab45528b5739a1a69f9c7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cdc525fcd31402f8d9dac64113ae636" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42395 } } peers { permanent_uuid: "cdc41b9889334d76a7018a9cac0e858f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39433 } } peers { permanent_uuid: "c4228f7905ab45528b5739a1a69f9c7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44609 } }
I20260926 20:48:51.123983 1442 raft_consensus.cc:374] T c0356eb5fa434477874da77246262c9d P c4228f7905ab45528b5739a1a69f9c7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:48:51.124027 1442 raft_consensus.cc:729] T c0356eb5fa434477874da77246262c9d P c4228f7905ab45528b5739a1a69f9c7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4228f7905ab45528b5739a1a69f9c7d, State: Initialized, Role: FOLLOWER
I20260926 20:48:51.124053 1444 log.cc:824] T c0356eb5fa434477874da77246262c9d P 8cdc525fcd31402f8d9dac64113ae636: Log is configured to *not* fsync() on all Append() calls
I20260926 20:48:51.124192 1442 consensus_queue.cc:260] T c0356eb5fa434477874da77246262c9d P c4228f7905ab45528b5739a1a69f9c7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cdc525fcd31402f8d9dac64113ae636" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42395 } } peers { permanent_uuid: "cdc41b9889334d76a7018a9cac0e858f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39433 } } peers { permanent_uuid: "c4228f7905ab45528b5739a1a69f9c7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44609 } }
I20260926 20:48:51.124392 1441 tablet_bootstrap.cc:654] T c0356eb5fa434477874da77246262c9d P cdc41b9889334d76a7018a9cac0e858f: Neither blocks nor log segments found. Creating new log.
I20260926 20:48:51.124465 1442 ts_tablet_manager.cc:1428] T c0356eb5fa434477874da77246262c9d P c4228f7905ab45528b5739a1a69f9c7d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:48:51.124773 1441 log.cc:824] T c0356eb5fa434477874da77246262c9d P cdc41b9889334d76a7018a9cac0e858f: Log is configured to *not* fsync() on all Append() calls
I20260926 20:48:51.124984 1444 tablet_bootstrap.cc:492] T c0356eb5fa434477874da77246262c9d P 8cdc525fcd31402f8d9dac64113ae636: No bootstrap required, opened a new log
I20260926 20:48:51.125052 1444 ts_tablet_manager.cc:1397] T c0356eb5fa434477874da77246262c9d P 8cdc525fcd31402f8d9dac64113ae636: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 20:48:51.125537 1441 tablet_bootstrap.cc:492] T c0356eb5fa434477874da77246262c9d P cdc41b9889334d76a7018a9cac0e858f: No bootstrap required, opened a new log
I20260926 20:48:51.125603 1441 ts_tablet_manager.cc:1397] T c0356eb5fa434477874da77246262c9d P cdc41b9889334d76a7018a9cac0e858f: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260926 20:48:51.126639 1444 raft_consensus.cc:348] T c0356eb5fa434477874da77246262c9d P 8cdc525fcd31402f8d9dac64113ae636 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cdc525fcd31402f8d9dac64113ae636" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42395 } } peers { permanent_uuid: "cdc41b9889334d76a7018a9cac0e858f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39433 } } peers { permanent_uuid: "c4228f7905ab45528b5739a1a69f9c7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44609 } }
I20260926 20:48:51.126757 1444 raft_consensus.cc:374] T c0356eb5fa434477874da77246262c9d P 8cdc525fcd31402f8d9dac64113ae636 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:48:51.126776 1444 raft_consensus.cc:729] T c0356eb5fa434477874da77246262c9d P 8cdc525fcd31402f8d9dac64113ae636 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cdc525fcd31402f8d9dac64113ae636, State: Initialized, Role: FOLLOWER
I20260926 20:48:51.126899 1444 consensus_queue.cc:260] T c0356eb5fa434477874da77246262c9d P 8cdc525fcd31402f8d9dac64113ae636 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cdc525fcd31402f8d9dac64113ae636" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42395 } } peers { permanent_uuid: "cdc41b9889334d76a7018a9cac0e858f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39433 } } peers { permanent_uuid: "c4228f7905ab45528b5739a1a69f9c7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44609 } }
I20260926 20:48:51.127170 1444 ts_tablet_manager.cc:1428] T c0356eb5fa434477874da77246262c9d P 8cdc525fcd31402f8d9dac64113ae636: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 20:48:51.127337 1422 heartbeater.cc:503] Master 127.0.98.62:40085 was elected leader, sending a full tablet report...
I20260926 20:48:51.127600 1441 raft_consensus.cc:348] T c0356eb5fa434477874da77246262c9d P cdc41b9889334d76a7018a9cac0e858f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cdc525fcd31402f8d9dac64113ae636" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42395 } } peers { permanent_uuid: "cdc41b9889334d76a7018a9cac0e858f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39433 } } peers { permanent_uuid: "c4228f7905ab45528b5739a1a69f9c7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44609 } }
I20260926 20:48:51.127781 1441 raft_consensus.cc:374] T c0356eb5fa434477874da77246262c9d P cdc41b9889334d76a7018a9cac0e858f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:48:51.127858 1441 raft_consensus.cc:729] T c0356eb5fa434477874da77246262c9d P cdc41b9889334d76a7018a9cac0e858f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdc41b9889334d76a7018a9cac0e858f, State: Initialized, Role: FOLLOWER
I20260926 20:48:51.127977 1441 consensus_queue.cc:260] T c0356eb5fa434477874da77246262c9d P cdc41b9889334d76a7018a9cac0e858f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cdc525fcd31402f8d9dac64113ae636" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42395 } } peers { permanent_uuid: "cdc41b9889334d76a7018a9cac0e858f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39433 } } peers { permanent_uuid: "c4228f7905ab45528b5739a1a69f9c7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44609 } }
I20260926 20:48:51.128257 1441 ts_tablet_manager.cc:1428] T c0356eb5fa434477874da77246262c9d P cdc41b9889334d76a7018a9cac0e858f: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
W20260926 20:48:51.138841 1154 tablet.cc:2369] T c0356eb5fa434477874da77246262c9d P c4228f7905ab45528b5739a1a69f9c7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:48:51.197988 1449 raft_consensus.cc:482] T c0356eb5fa434477874da77246262c9d P cdc41b9889334d76a7018a9cac0e858f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:48:51.198164 1449 raft_consensus.cc:504] T c0356eb5fa434477874da77246262c9d P cdc41b9889334d76a7018a9cac0e858f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cdc525fcd31402f8d9dac64113ae636" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42395 } } peers { permanent_uuid: "cdc41b9889334d76a7018a9cac0e858f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39433 } } peers { permanent_uuid: "c4228f7905ab45528b5739a1a69f9c7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44609 } }
I20260926 20:48:51.198596 1449 leader_election.cc:288] T c0356eb5fa434477874da77246262c9d P cdc41b9889334d76a7018a9cac0e858f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8cdc525fcd31402f8d9dac64113ae636 (127.0.98.3:42395), c4228f7905ab45528b5739a1a69f9c7d (127.0.98.1:44609)
I20260926 20:48:51.201711 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:51.198917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60563 (local address 127.0.98.1:44609)
0926 20:48:51.198965 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:51.198968 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:51.199112 (+ 144us) server_negotiation.cc:407] Connection header received
0926 20:48:51.199186 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:51.199187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:51.199200 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:51.199279 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:48:51.199450 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:51.199935 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:51.200977 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:51.201353 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:51.201444 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:51.201453 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:51.201611 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 20:48:51.201647 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 20:48:51.201712 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:51.198826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41049 (local address 127.0.98.3:42395)
0926 20:48:51.198877 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:51.198879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:51.199093 (+ 214us) server_negotiation.cc:407] Connection header received
0926 20:48:51.199128 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:51.199130 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:51.199143 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:51.199220 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:48:51.199392 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:51.199933 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:51.200972 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:51.201349 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:51.201439 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:51.201449 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:51.201610 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 20:48:51.201647 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 20:48:51.201730 1450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:51.198797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42395 (local address 127.0.98.2:41049)
0926 20:48:51.198959 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:51.198971 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:51.199043 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:51.199224 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:51.199225 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:51.199227 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:48:51.199315 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:51.199317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:51.200022 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:51.200023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:51.200872 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:51.200874 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:51.201440 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:51.201440 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:51.201522 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:51.201526 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:48:51.201593 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 20:48:51.201639 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":110,"threads_started":1}
I20260926 20:48:51.201735 1451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:51.198824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44609 (local address 127.0.98.2:60563)
0926 20:48:51.198990 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:51.199000 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:51.199129 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:51.199281 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:51.199282 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:51.199284 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:48:51.199372 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:51.199373 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:51.200033 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:51.200035 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:51.200872 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:51.200875 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:51.201454 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:51.201455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:51.201528 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:51.201531 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:48:51.201606 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 20:48:51.201647 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":57,"thread_start_us":104,"threads_started":1}
I20260926 20:48:51.202112 1376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0356eb5fa434477874da77246262c9d" candidate_uuid: "cdc41b9889334d76a7018a9cac0e858f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cdc525fcd31402f8d9dac64113ae636" is_pre_election: true
I20260926 20:48:51.202111 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0356eb5fa434477874da77246262c9d" candidate_uuid: "cdc41b9889334d76a7018a9cac0e858f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4228f7905ab45528b5739a1a69f9c7d" is_pre_election: true
I20260926 20:48:51.202278 1376 raft_consensus.cc:2454] T c0356eb5fa434477874da77246262c9d P 8cdc525fcd31402f8d9dac64113ae636 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdc41b9889334d76a7018a9cac0e858f in term 0.
I20260926 20:48:51.202279 1107 raft_consensus.cc:2454] T c0356eb5fa434477874da77246262c9d P c4228f7905ab45528b5739a1a69f9c7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdc41b9889334d76a7018a9cac0e858f in term 0.
I20260926 20:48:51.202548 1175 leader_election.cc:302] T c0356eb5fa434477874da77246262c9d P cdc41b9889334d76a7018a9cac0e858f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4228f7905ab45528b5739a1a69f9c7d, cdc41b9889334d76a7018a9cac0e858f; no voters:
I20260926 20:48:51.202723 1449 raft_consensus.cc:2781] T c0356eb5fa434477874da77246262c9d P cdc41b9889334d76a7018a9cac0e858f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:48:51.202785 1449 raft_consensus.cc:482] T c0356eb5fa434477874da77246262c9d P cdc41b9889334d76a7018a9cac0e858f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:48:51.202809 1449 raft_consensus.cc:3037] T c0356eb5fa434477874da77246262c9d P cdc41b9889334d76a7018a9cac0e858f [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:48:51.203723 1449 raft_consensus.cc:504] T c0356eb5fa434477874da77246262c9d P cdc41b9889334d76a7018a9cac0e858f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cdc525fcd31402f8d9dac64113ae636" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42395 } } peers { permanent_uuid: "cdc41b9889334d76a7018a9cac0e858f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39433 } } peers { permanent_uuid: "c4228f7905ab45528b5739a1a69f9c7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44609 } }
I20260926 20:48:51.203895 1449 leader_election.cc:288] T c0356eb5fa434477874da77246262c9d P cdc41b9889334d76a7018a9cac0e858f [CANDIDATE]: Term 1 election: Requested vote from peers 8cdc525fcd31402f8d9dac64113ae636 (127.0.98.3:42395), c4228f7905ab45528b5739a1a69f9c7d (127.0.98.1:44609)
I20260926 20:48:51.204103 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0356eb5fa434477874da77246262c9d" candidate_uuid: "cdc41b9889334d76a7018a9cac0e858f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4228f7905ab45528b5739a1a69f9c7d"
I20260926 20:48:51.204141 1376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0356eb5fa434477874da77246262c9d" candidate_uuid: "cdc41b9889334d76a7018a9cac0e858f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cdc525fcd31402f8d9dac64113ae636"
I20260926 20:48:51.204185 1107 raft_consensus.cc:3037] T c0356eb5fa434477874da77246262c9d P c4228f7905ab45528b5739a1a69f9c7d [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:48:51.204241 1376 raft_consensus.cc:3037] T c0356eb5fa434477874da77246262c9d P 8cdc525fcd31402f8d9dac64113ae636 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:48:51.204914 1107 raft_consensus.cc:2454] T c0356eb5fa434477874da77246262c9d P c4228f7905ab45528b5739a1a69f9c7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdc41b9889334d76a7018a9cac0e858f in term 1.
I20260926 20:48:51.205047 1376 raft_consensus.cc:2454] T c0356eb5fa434477874da77246262c9d P 8cdc525fcd31402f8d9dac64113ae636 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdc41b9889334d76a7018a9cac0e858f in term 1.
I20260926 20:48:51.205089 1175 leader_election.cc:302] T c0356eb5fa434477874da77246262c9d P cdc41b9889334d76a7018a9cac0e858f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4228f7905ab45528b5739a1a69f9c7d, cdc41b9889334d76a7018a9cac0e858f; no voters:
I20260926 20:48:51.205188 1449 raft_consensus.cc:2781] T c0356eb5fa434477874da77246262c9d P cdc41b9889334d76a7018a9cac0e858f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:48:51.205338 1449 raft_consensus.cc:686] T c0356eb5fa434477874da77246262c9d P cdc41b9889334d76a7018a9cac0e858f [term 1 LEADER]: Becoming Leader. State: Replica: cdc41b9889334d76a7018a9cac0e858f, State: Running, Role: LEADER
I20260926 20:48:51.205442 1449 consensus_queue.cc:237] T c0356eb5fa434477874da77246262c9d P cdc41b9889334d76a7018a9cac0e858f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cdc525fcd31402f8d9dac64113ae636" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42395 } } peers { permanent_uuid: "cdc41b9889334d76a7018a9cac0e858f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39433 } } peers { permanent_uuid: "c4228f7905ab45528b5739a1a69f9c7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44609 } }
I20260926 20:48:51.206269 961 catalog_manager.cc:5495] T c0356eb5fa434477874da77246262c9d P cdc41b9889334d76a7018a9cac0e858f reported cstate change: term changed from 0 to 1, leader changed from <none> to cdc41b9889334d76a7018a9cac0e858f (127.0.98.2). New cstate: current_term: 1 leader_uuid: "cdc41b9889334d76a7018a9cac0e858f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cdc525fcd31402f8d9dac64113ae636" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdc41b9889334d76a7018a9cac0e858f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4228f7905ab45528b5739a1a69f9c7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44609 } health_report { overall_health: UNKNOWN } } }
Sep 26 20:48:51 dist-test-slave-hwlj krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 20:48:51.236290 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:51.228217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44050 (local address 127.0.98.1:44609)
0926 20:48:51.228308 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:51.228311 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:51.228378 (+ 67us) server_negotiation.cc:407] Connection header received
0926 20:48:51.228456 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:51.228457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:51.228469 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:51.228570 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:51.229198 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:51.229685 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:51.230442 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:51.230646 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:51.230729 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:51.233217 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:51.233231 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:51.233231 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:51.233240 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:51.234938 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:51.235376 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:51.235380 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:51.235380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:51.235443 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:51.235665 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:51.235666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:51.235666 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:51.235880 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:51.236039 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:51.236102 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 20:48:51.236183 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 20:48:51 dist-test-slave-hwlj krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455731, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 20:48:51 dist-test-slave-hwlj krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455731, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 20:48:51.260605 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:51.252893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44058 (local address 127.0.98.1:44609)
0926 20:48:51.253004 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:51.253006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:51.253017 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:48:51.253094 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:51.253095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:51.253104 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:51.253156 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:51.253731 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:51.254265 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:51.255075 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:51.255291 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:51.255358 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:51.257969 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:51.257971 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:51.257972 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:51.257985 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:51.259267 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:51.259683 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:51.259685 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:51.259685 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:51.259788 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:51.259989 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:51.259990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:51.259991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:51.260104 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:51.260209 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:51.260408 (+ 199us) server_negotiation.cc:299] Negotiation successful
0926 20:48:51.260495 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260926 20:48:51.270541 1288 tablet.cc:2369] T c0356eb5fa434477874da77246262c9d P cdc41b9889334d76a7018a9cac0e858f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 20:48:51 dist-test-slave-hwlj krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 20:48:51 dist-test-slave-hwlj krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 20:48:51.285136 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:51.276654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44062 (local address 127.0.98.1:44609)
0926 20:48:51.276700 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:51.276703 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:51.276839 (+ 136us) server_negotiation.cc:407] Connection header received
0926 20:48:51.276890 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:51.276892 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:51.276919 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:51.276971 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:51.277581 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:51.278055 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:51.278817 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:51.279011 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:51.279092 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:51.281945 (+ 2853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:51.281946 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:51.281947 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:51.281954 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:51.283812 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:51.284224 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:51.284226 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:51.284226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:51.284295 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:51.284520 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:51.284521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:51.284521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:51.284654 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:51.284755 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:51.284936 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 20:48:51.285023 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 20:48:51.285907 1067 tablet_service.cc:2908] Rejecting scan request for tablet c0356eb5fa434477874da77246262c9d: Uninitialized: clean time has not yet been initialized
I20260926 20:48:51.286782 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 1024
I20260926 20:48:51.292351 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 1158
I20260926 20:48:51.296862 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 1293
I20260926 20:48:51.301533 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 930
2026-09-26T20:48:51Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3887 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:48:51 dist-test-slave-hwlj krb5kdc[1466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:48:51 dist-test-slave-hwlj krb5kdc[1466](info): set up 2 sockets
Sep 26 20:48:51 dist-test-slave-hwlj krb5kdc[1466](info): commencing operation
krb5kdc: starting...
W20260926 20:48:52.367501 392 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/krb5kdc/test-user.keytab.
Sep 26 20:48:52 dist-test-slave-hwlj krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:48:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:48:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 20:48:52.487144 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41023
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:48:52.566466 1479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:48:52.566737 1479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:48:52.566778 1479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:48:52.567965 1479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:48:52.568037 1479 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 20:48:52.568058 1479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:48:52.568082 1479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:48:52.568111 1479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:48:52.568130 1479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:48:52.569617 1479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41023
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:48:52.569892 1479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:48:52.570116 1479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:48:52.575007 1485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:48:52.575011 1484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:48:52.575321 1479 server_base.cc:1034] running on GCE node
W20260926 20:48:52.575057 1487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:48:52.575649 1479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:48:52.575973 1479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:48:52.577150 1479 hybrid_clock.cc:648] HybridClock initialized: now 1790455732577144 us; error 39 us; skew 500 ppm
Sep 26 20:48:52 dist-test-slave-hwlj krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455732, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:48:52.579682 1479 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:48:52.580142 1479 webserver.cc:460] Webserver started at http://127.0.98.62:39375/ using document root <none> and password file <none>
I20260926 20:48:52.580374 1479 fs_manager.cc:359] Metadata directory not provided
I20260926 20:48:52.580444 1479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:48:52.580600 1479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:48:52.581753 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/master-0/data/instance:
uuid: "6eb902a2976241ba9a38e69de9023356"
format_stamp: "Formatted at 2026-09-26 20:48:52 on dist-test-slave-hwlj"
server_key: "39dc917feb8212825ee9e19b4aca817a"
server_key_iv: "8a0ec54eb6cf0422832e4529978e0b97"
server_key_version: "encryptionkey@0"
I20260926 20:48:52.582123 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/master-0/wal/instance:
uuid: "6eb902a2976241ba9a38e69de9023356"
format_stamp: "Formatted at 2026-09-26 20:48:52 on dist-test-slave-hwlj"
server_key: "39dc917feb8212825ee9e19b4aca817a"
server_key_iv: "8a0ec54eb6cf0422832e4529978e0b97"
server_key_version: "encryptionkey@0"
I20260926 20:48:52.583523 1479 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:48:52.584498 1494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:48:52.584688 1479 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:48:52.584805 1479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "6eb902a2976241ba9a38e69de9023356"
format_stamp: "Formatted at 2026-09-26 20:48:52 on dist-test-slave-hwlj"
server_key: "39dc917feb8212825ee9e19b4aca817a"
server_key_iv: "8a0ec54eb6cf0422832e4529978e0b97"
server_key_version: "encryptionkey@0"
I20260926 20:48:52.584908 1479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:48:52.596496 1479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:48:52.598616 1479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:48:52.598804 1479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:48:52.604008 1479 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41023
I20260926 20:48:52.604059 1556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41023 every 8 connection(s)
I20260926 20:48:52.604416 1479 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:48:52.605108 1557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:48:52.607446 1557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6eb902a2976241ba9a38e69de9023356: Bootstrap starting.
I20260926 20:48:52.608095 1557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6eb902a2976241ba9a38e69de9023356: Neither blocks nor log segments found. Creating new log.
I20260926 20:48:52.608407 1557 log.cc:824] T 00000000000000000000000000000000 P 6eb902a2976241ba9a38e69de9023356: Log is configured to *not* fsync() on all Append() calls
I20260926 20:48:52.609210 1557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6eb902a2976241ba9a38e69de9023356: No bootstrap required, opened a new log
I20260926 20:48:52.611254 1557 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6eb902a2976241ba9a38e69de9023356 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eb902a2976241ba9a38e69de9023356" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41023 } }
I20260926 20:48:52.611433 1557 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6eb902a2976241ba9a38e69de9023356 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:48:52.611485 1557 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6eb902a2976241ba9a38e69de9023356 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6eb902a2976241ba9a38e69de9023356, State: Initialized, Role: FOLLOWER
I20260926 20:48:52.611631 1557 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6eb902a2976241ba9a38e69de9023356 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eb902a2976241ba9a38e69de9023356" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41023 } }
I20260926 20:48:52.611721 1557 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6eb902a2976241ba9a38e69de9023356 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:48:52.611809 1557 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6eb902a2976241ba9a38e69de9023356 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:48:52.611892 1557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6eb902a2976241ba9a38e69de9023356 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:48:52.611925 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 1479
I20260926 20:48:52.612011 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/master-0/wal/instance
I20260926 20:48:52.612136 392 external_mini_cluster.cc:1428] Setting key 13f6bb55c1a838a874c3cbb160e0ab50
I20260926 20:48:52.612671 1557 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6eb902a2976241ba9a38e69de9023356 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eb902a2976241ba9a38e69de9023356" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41023 } }
I20260926 20:48:52.612779 1557 leader_election.cc:302] T 00000000000000000000000000000000 P 6eb902a2976241ba9a38e69de9023356 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6eb902a2976241ba9a38e69de9023356; no voters:
I20260926 20:48:52.612963 1557 leader_election.cc:288] T 00000000000000000000000000000000 P 6eb902a2976241ba9a38e69de9023356 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:48:52.613094 1562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6eb902a2976241ba9a38e69de9023356 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:48:52.613245 1557 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6eb902a2976241ba9a38e69de9023356 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:48:52.613232 1562 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6eb902a2976241ba9a38e69de9023356 [term 1 LEADER]: Becoming Leader. State: Replica: 6eb902a2976241ba9a38e69de9023356, State: Running, Role: LEADER
I20260926 20:48:52.613569 1562 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6eb902a2976241ba9a38e69de9023356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eb902a2976241ba9a38e69de9023356" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41023 } }
I20260926 20:48:52.613914 1562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6eb902a2976241ba9a38e69de9023356 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6eb902a2976241ba9a38e69de9023356. Latest consensus state: current_term: 1 leader_uuid: "6eb902a2976241ba9a38e69de9023356" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eb902a2976241ba9a38e69de9023356" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41023 } } }
I20260926 20:48:52.613991 1562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6eb902a2976241ba9a38e69de9023356 [sys.catalog]: This master's current role is: LEADER
I20260926 20:48:52.614365 1563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6eb902a2976241ba9a38e69de9023356 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6eb902a2976241ba9a38e69de9023356" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eb902a2976241ba9a38e69de9023356" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41023 } } }
I20260926 20:48:52.614451 1563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6eb902a2976241ba9a38e69de9023356 [sys.catalog]: This master's current role is: LEADER
W20260926 20:48:52.615079 1577 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6eb902a2976241ba9a38e69de9023356: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:48:52.615154 1577 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:48:52.615218 1571 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:48:52.615361 1571 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:48:52.617014 1571 catalog_manager.cc:1347] Generated new cluster ID: cb012c04dad847679db90c608f014526
I20260926 20:48:52.617070 1571 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 20:48:52 dist-test-slave-hwlj krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:48:52.622915 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:52.612703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48160 (local address 127.0.98.62:41023)
0926 20:48:52.612929 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:52.612939 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:52.612974 (+ 35us) server_negotiation.cc:407] Connection header received
0926 20:48:52.613094 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:52.613101 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:52.613169 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:52.613313 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:52.614142 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:52.615607 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:52.616571 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:52.617053 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:52.617183 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:52.619581 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:52.619585 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:52.619586 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:52.619603 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:52.621332 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:52.621748 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:52.621751 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:52.621752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:52.621857 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:52.622078 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:52.622079 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:52.622079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:52.622311 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:52.622510 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:52.622586 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 20:48:52.622747 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":93,"threads_started":1}
I20260926 20:48:52.665057 1571 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:48:52.665665 1571 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:48:52.670482 1571 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6eb902a2976241ba9a38e69de9023356: Generated new TSK 0
I20260926 20:48:52.670706 1571 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 20:48:52.707101 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41023
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:48:52.789489 1583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:48:52.789716 1583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:48:52.789757 1583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:48:52.790896 1583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:48:52.790963 1583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:48:52.791002 1583 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:48:52.791059 1583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:48:52.792634 1583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:41023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:48:52.792900 1583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:48:52.793125 1583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:48:52.798143 1591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:48:52.798187 1589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:48:52.798216 1583 server_base.cc:1034] running on GCE node
W20260926 20:48:52.798187 1588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:48:52.798518 1583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:48:52.798779 1583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:48:52.799944 1583 hybrid_clock.cc:648] HybridClock initialized: now 1790455732799926 us; error 30 us; skew 500 ppm
Sep 26 20:48:52 dist-test-slave-hwlj krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455732, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:48:52.802412 1583 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 20:48:52.802857 1583 webserver.cc:460] Webserver started at http://127.0.98.1:36097/ using document root <none> and password file <none>
I20260926 20:48:52.803033 1583 fs_manager.cc:359] Metadata directory not provided
I20260926 20:48:52.803072 1583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:48:52.803160 1583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:48:52.804217 1583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-0/data/instance:
uuid: "0f844a7566be4416bc8dc3f9cb375536"
format_stamp: "Formatted at 2026-09-26 20:48:52 on dist-test-slave-hwlj"
server_key: "1acd7c23953230bad7b8bc1b6ede1f30"
server_key_iv: "2dd900317167f9692d70ff13aded8dc5"
server_key_version: "encryptionkey@0"
I20260926 20:48:52.804523 1583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0f844a7566be4416bc8dc3f9cb375536"
format_stamp: "Formatted at 2026-09-26 20:48:52 on dist-test-slave-hwlj"
server_key: "1acd7c23953230bad7b8bc1b6ede1f30"
server_key_iv: "2dd900317167f9692d70ff13aded8dc5"
server_key_version: "encryptionkey@0"
I20260926 20:48:52.805787 1583 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:48:52.806521 1598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:48:52.806775 1583 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:48:52.806864 1583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-0/wal
uuid: "0f844a7566be4416bc8dc3f9cb375536"
format_stamp: "Formatted at 2026-09-26 20:48:52 on dist-test-slave-hwlj"
server_key: "1acd7c23953230bad7b8bc1b6ede1f30"
server_key_iv: "2dd900317167f9692d70ff13aded8dc5"
server_key_version: "encryptionkey@0"
I20260926 20:48:52.806919 1583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:48:52.815310 1583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:48:52.817625 1583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:48:52.817739 1583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:48:52.818188 1583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:48:52.818223 1583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:48:52.818245 1583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:48:52.818266 1583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:48:52.824995 1583 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43301
I20260926 20:48:52.825124 1713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43301 every 8 connection(s)
I20260926 20:48:52.825402 1583 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 20:48:52 dist-test-slave-hwlj krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455732, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:48:52.829258 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:52.818732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33863 (local address 127.0.98.62:41023)
0926 20:48:52.818807 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:52.818810 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:52.818987 (+ 177us) server_negotiation.cc:407] Connection header received
0926 20:48:52.819489 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:52.819490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:52.819507 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:52.819588 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:52.820541 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:52.821416 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:52.823061 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:52.823397 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:52.823488 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:52.826406 (+ 2918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:52.826407 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:52.826407 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:52.826414 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:52.827701 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:52.828225 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:52.828226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:52.828226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:52.828278 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:52.828591 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:52.828593 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:52.828593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:52.828720 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:52.828791 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:52.829120 (+ 329us) server_negotiation.cc:299] Negotiation successful
0926 20:48:52.829196 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 20:48:52.829306 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:52.818684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41023 (local address 127.0.98.1:33863)
0926 20:48:52.818858 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:52.818899 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:52.819391 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:52.819726 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:52.819730 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:52.820136 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:48:52.820427 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:52.820434 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:52.821534 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:52.821535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:52.822952 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:52.822955 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:52.823562 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:52.823564 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:52.823742 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:52.824465 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:48:52.824475 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:48:52.826314 (+ 1839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:48:52.827837 (+ 1523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:52.827844 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:52.827846 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:52.828124 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:52.828366 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:52.828367 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:52.828367 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:52.828485 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:52.828814 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:48:52.828820 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:48:52.828934 (+ 114us) client_negotiation.cc:769] Sending connection context
0926 20:48:52.829028 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 20:48:52.829155 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":89,"threads_started":1}
I20260926 20:48:52.829909 1714 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41023
I20260926 20:48:52.829994 1714 heartbeater.cc:464] Registering TS with master...
I20260926 20:48:52.830211 1714 heartbeater.cc:511] Master 127.0.98.62:41023 requested a full tablet report, sending...
I20260926 20:48:52.830861 1511 ts_manager.cc:194] Registered new tserver with Master: 0f844a7566be4416bc8dc3f9cb375536 (127.0.98.1:43301)
I20260926 20:48:52.831708 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33863
I20260926 20:48:52.832619 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 1583
I20260926 20:48:52.832715 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-0/wal/instance
I20260926 20:48:52.832872 392 external_mini_cluster.cc:1428] Setting key 30e75609bf181a90fd92963144f4351a
I20260926 20:48:52.837146 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:52.831025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58823 (local address 127.0.98.62:41023)
0926 20:48:52.831064 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:52.831066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:52.831206 (+ 140us) server_negotiation.cc:407] Connection header received
0926 20:48:52.831227 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:52.831228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:52.831238 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:52.831295 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:52.831570 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:52.832251 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:52.833244 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:52.833482 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:52.833539 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:52.834502 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:52.834503 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:52.834504 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:52.834512 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:52.835973 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:52.836344 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:52.836345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:52.836346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:52.836402 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:52.836621 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:52.836622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:52.836622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:52.836737 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:52.836817 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:52.836980 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 20:48:52.837058 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 20:48:52.837162 1720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:52.830978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41023 (local address 127.0.98.1:58823)
0926 20:48:52.831098 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:52.831108 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:52.831166 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:52.831308 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:52.831309 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:52.831451 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:48:52.831504 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:52.831505 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:52.832360 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:52.832361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:52.833149 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:52.833152 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:52.833561 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:52.833562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:52.833630 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:52.834105 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:48:52.834110 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:48:52.834411 (+ 301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:48:52.836076 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:52.836077 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:52.836078 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:52.836252 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:52.836471 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:52.836472 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:52.836473 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:52.836544 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:52.836808 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:48:52.836809 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:48:52.836866 (+ 57us) client_negotiation.cc:769] Sending connection context
0926 20:48:52.836955 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 20:48:52.837059 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":63,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 20:48:52.860040 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41023
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:48:52.940907 1723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:48:52.941125 1723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:48:52.941150 1723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:48:52.942308 1723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:48:52.942354 1723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:48:52.942375 1723 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:48:52.942394 1723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:48:52.943894 1723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:41023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:48:52.944101 1723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:48:52.944358 1723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:48:52.949725 1729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:48:52.949759 1731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:48:52.949725 1728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:48:52.950002 1723 server_base.cc:1034] running on GCE node
I20260926 20:48:52.950327 1723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:48:52.950595 1723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:48:52.951874 1723 hybrid_clock.cc:648] HybridClock initialized: now 1790455732951860 us; error 34 us; skew 500 ppm
Sep 26 20:48:52 dist-test-slave-hwlj krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455732, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:48:52.954622 1723 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 20:48:52.955132 1723 webserver.cc:460] Webserver started at http://127.0.98.2:44323/ using document root <none> and password file <none>
I20260926 20:48:52.955390 1723 fs_manager.cc:359] Metadata directory not provided
I20260926 20:48:52.955468 1723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:48:52.955632 1723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:48:52.956842 1723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-1/data/instance:
uuid: "6e3aa0a8326e4168add03993b7255812"
format_stamp: "Formatted at 2026-09-26 20:48:52 on dist-test-slave-hwlj"
server_key: "84514f38d03b3ec130cd989c42edec44"
server_key_iv: "6e93af7a4866bb6c1e0ac33028fba8aa"
server_key_version: "encryptionkey@0"
I20260926 20:48:52.957298 1723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6e3aa0a8326e4168add03993b7255812"
format_stamp: "Formatted at 2026-09-26 20:48:52 on dist-test-slave-hwlj"
server_key: "84514f38d03b3ec130cd989c42edec44"
server_key_iv: "6e93af7a4866bb6c1e0ac33028fba8aa"
server_key_version: "encryptionkey@0"
I20260926 20:48:52.958734 1723 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:48:52.959619 1738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:48:52.959875 1723 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:48:52.960027 1723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-1/wal
uuid: "6e3aa0a8326e4168add03993b7255812"
format_stamp: "Formatted at 2026-09-26 20:48:52 on dist-test-slave-hwlj"
server_key: "84514f38d03b3ec130cd989c42edec44"
server_key_iv: "6e93af7a4866bb6c1e0ac33028fba8aa"
server_key_version: "encryptionkey@0"
I20260926 20:48:52.960115 1723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:48:52.983101 1723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:48:52.985294 1723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:48:52.985466 1723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:48:52.986079 1723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:48:52.986158 1723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:48:52.986197 1723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:48:52.986224 1723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 20:48:52 dist-test-slave-hwlj krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455732, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:48:52.993716 1723 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44837
I20260926 20:48:52.993803 1853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44837 every 8 connection(s)
I20260926 20:48:52.994127 1723 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
I20260926 20:48:52.995078 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 1723
I20260926 20:48:52.995151 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-1/wal/instance
I20260926 20:48:52.995349 392 external_mini_cluster.cc:1428] Setting key ae7b6512fa1114eb1ae7b2b668c7c66e
I20260926 20:48:52.996492 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:52.986618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41717 (local address 127.0.98.62:41023)
0926 20:48:52.986697 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:52.986700 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:52.986969 (+ 269us) server_negotiation.cc:407] Connection header received
0926 20:48:52.987783 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:52.987785 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:52.987805 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:52.987881 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:52.988828 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:52.989394 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:52.990447 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:52.990657 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:52.990739 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:52.993012 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:52.993014 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:52.993015 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:52.993023 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:52.994927 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:52.995419 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:52.995420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:52.995420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:52.995478 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:52.995783 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:52.995784 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:52.995784 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:52.995903 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:52.995986 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:52.996327 (+ 341us) server_negotiation.cc:299] Negotiation successful
0926 20:48:52.996404 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 20:48:52.996695 1747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:52.986602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41023 (local address 127.0.98.2:41717)
0926 20:48:52.986813 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:52.986864 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:52.987605 (+ 741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:52.987989 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:52.987993 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:52.988397 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:48:52.988724 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:52.988732 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:52.989494 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:52.989496 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:52.990328 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:52.990330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:52.990762 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:52.990763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:52.990905 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:52.991434 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:48:52.991444 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:48:52.992932 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:48:52.995025 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:52.995031 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:52.995033 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:52.995335 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:52.995556 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:52.995557 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:52.995557 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:52.995683 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:52.996037 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:48:52.996042 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:48:52.996177 (+ 135us) client_negotiation.cc:769] Sending connection context
0926 20:48:52.996292 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 20:48:52.996448 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":100,"threads_started":1}
I20260926 20:48:52.997175 1854 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41023
I20260926 20:48:52.997303 1854 heartbeater.cc:464] Registering TS with master...
I20260926 20:48:52.997548 1854 heartbeater.cc:511] Master 127.0.98.62:41023 requested a full tablet report, sending...
I20260926 20:48:52.997936 1510 ts_manager.cc:194] Registered new tserver with Master: 6e3aa0a8326e4168add03993b7255812 (127.0.98.2:44837)
I20260926 20:48:52.998342 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41717
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 20:48:53.006996 1861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:52.999997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41023 (local address 127.0.98.2:37477)
0926 20:48:53.000229 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:53.000239 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:53.000294 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:53.000431 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:53.000432 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:53.000618 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:48:53.000694 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:53.000695 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.001707 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.001708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:53.002273 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:53.002275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.002794 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.002795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:53.002860 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:53.003379 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:48:53.003384 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:48:53.003735 (+ 351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:48:53.005764 (+ 2029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:53.005765 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:53.005765 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:53.006021 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:53.006290 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:53.006291 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:53.006291 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:53.006347 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:53.006620 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:48:53.006621 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:48:53.006701 (+ 80us) client_negotiation.cc:769] Sending connection context
0926 20:48:53.006799 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 20:48:53.006889 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":139,"threads_started":1}
I20260926 20:48:53.007131 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:53.000100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37477 (local address 127.0.98.62:41023)
0926 20:48:53.000169 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:53.000171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:53.000307 (+ 136us) server_negotiation.cc:407] Connection header received
0926 20:48:53.000351 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:53.000352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:53.000364 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:53.000428 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:53.000769 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.001581 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.002371 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.002673 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.002755 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:53.003922 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:53.003923 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:53.003924 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:53.003932 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:53.005517 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:53.006157 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:53.006158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:53.006159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:53.006222 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:53.006408 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:53.006409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:53.006409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:53.006554 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:53.006615 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:53.006872 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 20:48:53.006958 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 20:48:53.023496 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41023
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:48:53.104220 1863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:48:53.104419 1863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:48:53.104442 1863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:48:53.105615 1863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:48:53.105660 1863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:48:53.105681 1863 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:48:53.105700 1863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:48:53.107239 1863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:41023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:48:53.107493 1863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:48:53.107813 1863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:48:53.113235 1871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:48:53.113293 1863 server_base.cc:1034] running on GCE node
W20260926 20:48:53.113261 1869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:48:53.113269 1868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:48:53.113631 1863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:48:53.114012 1863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:48:53.115240 1863 hybrid_clock.cc:648] HybridClock initialized: now 1790455733115229 us; error 47 us; skew 500 ppm
Sep 26 20:48:53 dist-test-slave-hwlj krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455733, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:48:53.117973 1863 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 20:48:53.118446 1863 webserver.cc:460] Webserver started at http://127.0.98.3:44661/ using document root <none> and password file <none>
I20260926 20:48:53.118680 1863 fs_manager.cc:359] Metadata directory not provided
I20260926 20:48:53.118755 1863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:48:53.118914 1863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:48:53.120056 1863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-2/data/instance:
uuid: "c1d2eebba7c14a698ed9ba4d04bb50d3"
format_stamp: "Formatted at 2026-09-26 20:48:53 on dist-test-slave-hwlj"
server_key: "c24569f6cff99f0a7908ee07807718a8"
server_key_iv: "008871d8aaceace8a1a5addde0437b0c"
server_key_version: "encryptionkey@0"
I20260926 20:48:53.120432 1863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c1d2eebba7c14a698ed9ba4d04bb50d3"
format_stamp: "Formatted at 2026-09-26 20:48:53 on dist-test-slave-hwlj"
server_key: "c24569f6cff99f0a7908ee07807718a8"
server_key_iv: "008871d8aaceace8a1a5addde0437b0c"
server_key_version: "encryptionkey@0"
I20260926 20:48:53.121796 1863 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:48:53.122646 1878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:48:53.122879 1863 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:48:53.122985 1863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-2/wal
uuid: "c1d2eebba7c14a698ed9ba4d04bb50d3"
format_stamp: "Formatted at 2026-09-26 20:48:53 on dist-test-slave-hwlj"
server_key: "c24569f6cff99f0a7908ee07807718a8"
server_key_iv: "008871d8aaceace8a1a5addde0437b0c"
server_key_version: "encryptionkey@0"
I20260926 20:48:53.123061 1863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:48:53.137723 1863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:48:53.140071 1863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:48:53.140192 1863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:48:53.140673 1863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:48:53.140708 1863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:48:53.140730 1863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:48:53.140743 1863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:48:53.148121 1863 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45635
I20260926 20:48:53.148175 1993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45635 every 8 connection(s)
Sep 26 20:48:53 dist-test-slave-hwlj krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455733, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:48:53.148500 1863 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
I20260926 20:48:53.148705 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 1863
I20260926 20:48:53.148774 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-2/wal/instance
I20260926 20:48:53.148939 392 external_mini_cluster.cc:1428] Setting key e86f43dce5d3b5205322c42daa5d3282
I20260926 20:48:53.151639 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:53.141332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33979 (local address 127.0.98.62:41023)
0926 20:48:53.141412 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:53.141414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:53.141657 (+ 243us) server_negotiation.cc:407] Connection header received
0926 20:48:53.142448 (+ 791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:53.142450 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:53.142463 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:53.142523 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:53.143461 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.144200 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.145917 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.146139 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.146199 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:53.149051 (+ 2852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:53.149053 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:53.149053 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:53.149059 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:53.150515 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:53.150878 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:53.150878 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:53.150878 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:53.150936 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:53.151163 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:53.151164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:53.151164 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:53.151263 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:53.151313 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:53.151521 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 20:48:53.151578 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 20:48:53.151912 1887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:53.141304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41023 (local address 127.0.98.3:33979)
0926 20:48:53.141504 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:53.141558 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:53.142305 (+ 747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:53.142619 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:53.142624 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:53.143050 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:48:53.143352 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:53.143359 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.144280 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.144282 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:53.145820 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:53.145825 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.146230 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.146232 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:53.146386 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:53.147061 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:48:53.147072 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:48:53.148947 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:48:53.150584 (+ 1637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:53.150589 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:53.150591 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:53.150810 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:53.150994 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:53.150995 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:53.150995 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:53.151104 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:53.151319 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:48:53.151323 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:48:53.151430 (+ 107us) client_negotiation.cc:769] Sending connection context
0926 20:48:53.151509 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 20:48:53.151631 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":96,"threads_started":1}
I20260926 20:48:53.152396 1994 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41023
I20260926 20:48:53.152498 1994 heartbeater.cc:464] Registering TS with master...
I20260926 20:48:53.152668 1994 heartbeater.cc:511] Master 127.0.98.62:41023 requested a full tablet report, sending...
I20260926 20:48:53.153195 1511 ts_manager.cc:194] Registered new tserver with Master: c1d2eebba7c14a698ed9ba4d04bb50d3 (127.0.98.3:45635)
I20260926 20:48:53.153707 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33979
I20260926 20:48:53.158779 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:53.153501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53627 (local address 127.0.98.62:41023)
0926 20:48:53.153538 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:53.153540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:53.153592 (+ 52us) server_negotiation.cc:407] Connection header received
0926 20:48:53.153655 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:53.153656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:53.153664 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:53.153727 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:53.154088 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.154551 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.155265 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.155465 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.155543 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:53.156457 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:53.156459 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:53.156459 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:53.156465 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:53.157649 (+ 1184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:53.158004 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:53.158005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:53.158006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:53.158070 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:53.158271 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:53.158273 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:53.158273 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:53.158381 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:53.158459 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:53.158612 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 20:48:53.158690 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 20:48:53.158799 2000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:53.153404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41023 (local address 127.0.98.3:53627)
0926 20:48:53.153538 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:53.153549 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:53.153605 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:53.153736 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:53.153737 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:53.153920 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:48:53.153992 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:53.153993 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.154651 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.154652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:53.155165 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:53.155166 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.155554 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.155555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:53.155618 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:53.156076 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:48:53.156080 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:48:53.156348 (+ 268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:48:53.157750 (+ 1402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:53.157751 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:53.157751 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:53.157916 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:53.158139 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:53.158140 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:53.158140 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:53.158196 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:53.158451 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:48:53.158452 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:48:53.158494 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 20:48:53.158588 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 20:48:53.158701 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":79,"threads_started":1}
I20260926 20:48:53.161028 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 20:48:53 dist-test-slave-hwlj krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455733, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 20:48:53 dist-test-slave-hwlj krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455733, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:48:53.183636 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:53.176257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48166 (local address 127.0.98.62:41023)
0926 20:48:53.176332 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:53.176334 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:53.176352 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:48:53.176376 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:53.176377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:53.176385 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:53.176439 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:53.177107 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.177602 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.178413 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.178612 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.178675 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:53.180955 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:53.180957 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:53.180957 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:53.180963 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:53.182349 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:53.182715 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:53.182716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:53.182717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:53.182772 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:53.182966 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:53.182967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:53.182967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:53.183087 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:53.183163 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:53.183368 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 20:48:53.183498 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 20:48:53.185364 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48166:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:48:53.186908 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:48:53.193444 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:53.189296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44837 (local address 127.0.0.1:57546)
0926 20:48:53.189473 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:53.189521 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:53.189588 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:53.189937 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:53.189941 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:53.189944 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:48:53.190038 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:53.190040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.191038 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.191039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:53.192286 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:53.192288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.193145 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.193145 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:53.193220 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:53.193233 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 20:48:53.193308 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 20:48:53.193357 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":1,"thread_start_us":110,"threads_started":1}
I20260926 20:48:53.193450 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:53.189556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57546 (local address 127.0.98.2:44837)
0926 20:48:53.189741 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:53.189746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:53.189757 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:48:53.189801 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:53.189804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:53.189821 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:53.189910 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:48:53.190139 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.190934 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.192378 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.193036 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.193143 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:53.193222 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:53.193322 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 20:48:53.193357 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":92,"threads_started":1}
I20260926 20:48:53.193711 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:53.189298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45635 (local address 127.0.0.1:50854)
0926 20:48:53.189680 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:53.189687 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:53.189720 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:53.190130 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:53.190132 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:53.190134 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:48:53.190196 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:53.190198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.191474 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.191476 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:53.192543 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:53.192545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.193484 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.193485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:53.193552 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:53.193558 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 20:48:53.193589 (+ 31us) client_negotiation.cc:240] Negotiation successful
0926 20:48:53.193626 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":2,"thread_start_us":81,"threads_started":1}
I20260926 20:48:53.193918 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:53.189298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43301 (local address 127.0.0.1:36666)
0926 20:48:53.189480 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:53.189521 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:53.189588 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:53.190357 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:53.190357 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:53.190359 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:48:53.190390 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:53.190391 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.191605 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.191606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:53.192704 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:53.192707 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.193596 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.193596 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:53.193645 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:53.193647 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 20:48:53.193675 (+ 28us) client_negotiation.cc:240] Negotiation successful
0926 20:48:53.193704 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":62,"threads_started":1}
I20260926 20:48:53.194094 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:53.189549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36666 (local address 127.0.98.1:43301)
0926 20:48:53.190053 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:53.190057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:53.190200 (+ 143us) server_negotiation.cc:407] Connection header received
0926 20:48:53.190242 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:53.190244 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:53.190261 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:53.190454 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:48:53.190509 (+ 55us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.191516 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.192798 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.193508 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.193755 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:53.193844 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:53.193872 (+ 28us) server_negotiation.cc:299] Negotiation successful
0926 20:48:53.193916 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":86,"threads_started":1}
I20260926 20:48:53.194173 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:53.189738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50854 (local address 127.0.98.3:45635)
0926 20:48:53.189920 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:53.189925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:53.189934 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:48:53.189969 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:53.189971 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:53.190042 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:53.190117 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:48:53.190286 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.191374 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.192633 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.193376 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.193910 (+ 534us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:53.194000 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:53.194026 (+ 26us) server_negotiation.cc:299] Negotiation successful
0926 20:48:53.194072 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":88,"threads_started":1}
I20260926 20:48:53.194491 1788 tablet_service.cc:1467] Processing CreateTablet for tablet b20b52a040fb45a0a32100756afe7a81 (DEFAULT_TABLE table=test-table [id=457671e304ca4390a5389f77d7c0a49d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:48:53.194681 1648 tablet_service.cc:1467] Processing CreateTablet for tablet b20b52a040fb45a0a32100756afe7a81 (DEFAULT_TABLE table=test-table [id=457671e304ca4390a5389f77d7c0a49d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:48:53.194867 1788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b20b52a040fb45a0a32100756afe7a81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:48:53.194769 1928 tablet_service.cc:1467] Processing CreateTablet for tablet b20b52a040fb45a0a32100756afe7a81 (DEFAULT_TABLE table=test-table [id=457671e304ca4390a5389f77d7c0a49d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:48:53.194929 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b20b52a040fb45a0a32100756afe7a81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:48:53.195072 1928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b20b52a040fb45a0a32100756afe7a81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:48:53.198215 2017 tablet_bootstrap.cc:492] T b20b52a040fb45a0a32100756afe7a81 P 0f844a7566be4416bc8dc3f9cb375536: Bootstrap starting.
I20260926 20:48:53.198956 2018 tablet_bootstrap.cc:492] T b20b52a040fb45a0a32100756afe7a81 P c1d2eebba7c14a698ed9ba4d04bb50d3: Bootstrap starting.
I20260926 20:48:53.199002 2017 tablet_bootstrap.cc:654] T b20b52a040fb45a0a32100756afe7a81 P 0f844a7566be4416bc8dc3f9cb375536: Neither blocks nor log segments found. Creating new log.
I20260926 20:48:53.199321 2017 log.cc:824] T b20b52a040fb45a0a32100756afe7a81 P 0f844a7566be4416bc8dc3f9cb375536: Log is configured to *not* fsync() on all Append() calls
I20260926 20:48:53.199550 2018 tablet_bootstrap.cc:654] T b20b52a040fb45a0a32100756afe7a81 P c1d2eebba7c14a698ed9ba4d04bb50d3: Neither blocks nor log segments found. Creating new log.
I20260926 20:48:53.199908 2019 tablet_bootstrap.cc:492] T b20b52a040fb45a0a32100756afe7a81 P 6e3aa0a8326e4168add03993b7255812: Bootstrap starting.
I20260926 20:48:53.199968 2018 log.cc:824] T b20b52a040fb45a0a32100756afe7a81 P c1d2eebba7c14a698ed9ba4d04bb50d3: Log is configured to *not* fsync() on all Append() calls
I20260926 20:48:53.200270 2017 tablet_bootstrap.cc:492] T b20b52a040fb45a0a32100756afe7a81 P 0f844a7566be4416bc8dc3f9cb375536: No bootstrap required, opened a new log
I20260926 20:48:53.200354 2017 ts_tablet_manager.cc:1397] T b20b52a040fb45a0a32100756afe7a81 P 0f844a7566be4416bc8dc3f9cb375536: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:48:53.200623 2019 tablet_bootstrap.cc:654] T b20b52a040fb45a0a32100756afe7a81 P 6e3aa0a8326e4168add03993b7255812: Neither blocks nor log segments found. Creating new log.
I20260926 20:48:53.200739 2018 tablet_bootstrap.cc:492] T b20b52a040fb45a0a32100756afe7a81 P c1d2eebba7c14a698ed9ba4d04bb50d3: No bootstrap required, opened a new log
I20260926 20:48:53.200809 2018 ts_tablet_manager.cc:1397] T b20b52a040fb45a0a32100756afe7a81 P c1d2eebba7c14a698ed9ba4d04bb50d3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 20:48:53.201042 2019 log.cc:824] T b20b52a040fb45a0a32100756afe7a81 P 6e3aa0a8326e4168add03993b7255812: Log is configured to *not* fsync() on all Append() calls
I20260926 20:48:53.201885 2019 tablet_bootstrap.cc:492] T b20b52a040fb45a0a32100756afe7a81 P 6e3aa0a8326e4168add03993b7255812: No bootstrap required, opened a new log
I20260926 20:48:53.201979 2019 ts_tablet_manager.cc:1397] T b20b52a040fb45a0a32100756afe7a81 P 6e3aa0a8326e4168add03993b7255812: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:48:53.202406 2017 raft_consensus.cc:348] T b20b52a040fb45a0a32100756afe7a81 P 0f844a7566be4416bc8dc3f9cb375536 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d2eebba7c14a698ed9ba4d04bb50d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45635 } } peers { permanent_uuid: "6e3aa0a8326e4168add03993b7255812" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44837 } } peers { permanent_uuid: "0f844a7566be4416bc8dc3f9cb375536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43301 } }
I20260926 20:48:53.202593 2017 raft_consensus.cc:374] T b20b52a040fb45a0a32100756afe7a81 P 0f844a7566be4416bc8dc3f9cb375536 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:48:53.202631 2017 raft_consensus.cc:729] T b20b52a040fb45a0a32100756afe7a81 P 0f844a7566be4416bc8dc3f9cb375536 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f844a7566be4416bc8dc3f9cb375536, State: Initialized, Role: FOLLOWER
I20260926 20:48:53.202780 2017 consensus_queue.cc:260] T b20b52a040fb45a0a32100756afe7a81 P 0f844a7566be4416bc8dc3f9cb375536 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d2eebba7c14a698ed9ba4d04bb50d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45635 } } peers { permanent_uuid: "6e3aa0a8326e4168add03993b7255812" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44837 } } peers { permanent_uuid: "0f844a7566be4416bc8dc3f9cb375536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43301 } }
I20260926 20:48:53.202765 2018 raft_consensus.cc:348] T b20b52a040fb45a0a32100756afe7a81 P c1d2eebba7c14a698ed9ba4d04bb50d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d2eebba7c14a698ed9ba4d04bb50d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45635 } } peers { permanent_uuid: "6e3aa0a8326e4168add03993b7255812" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44837 } } peers { permanent_uuid: "0f844a7566be4416bc8dc3f9cb375536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43301 } }
I20260926 20:48:53.202922 2018 raft_consensus.cc:374] T b20b52a040fb45a0a32100756afe7a81 P c1d2eebba7c14a698ed9ba4d04bb50d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:48:53.202962 2018 raft_consensus.cc:729] T b20b52a040fb45a0a32100756afe7a81 P c1d2eebba7c14a698ed9ba4d04bb50d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1d2eebba7c14a698ed9ba4d04bb50d3, State: Initialized, Role: FOLLOWER
I20260926 20:48:53.203064 2017 ts_tablet_manager.cc:1428] T b20b52a040fb45a0a32100756afe7a81 P 0f844a7566be4416bc8dc3f9cb375536: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:48:53.203089 2018 consensus_queue.cc:260] T b20b52a040fb45a0a32100756afe7a81 P c1d2eebba7c14a698ed9ba4d04bb50d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d2eebba7c14a698ed9ba4d04bb50d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45635 } } peers { permanent_uuid: "6e3aa0a8326e4168add03993b7255812" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44837 } } peers { permanent_uuid: "0f844a7566be4416bc8dc3f9cb375536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43301 } }
I20260926 20:48:53.203205 1714 heartbeater.cc:503] Master 127.0.98.62:41023 was elected leader, sending a full tablet report...
I20260926 20:48:53.203346 2018 ts_tablet_manager.cc:1428] T b20b52a040fb45a0a32100756afe7a81 P c1d2eebba7c14a698ed9ba4d04bb50d3: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260926 20:48:53.203353 1994 heartbeater.cc:503] Master 127.0.98.62:41023 was elected leader, sending a full tablet report...
I20260926 20:48:53.204001 2019 raft_consensus.cc:348] T b20b52a040fb45a0a32100756afe7a81 P 6e3aa0a8326e4168add03993b7255812 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d2eebba7c14a698ed9ba4d04bb50d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45635 } } peers { permanent_uuid: "6e3aa0a8326e4168add03993b7255812" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44837 } } peers { permanent_uuid: "0f844a7566be4416bc8dc3f9cb375536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43301 } }
I20260926 20:48:53.204177 2019 raft_consensus.cc:374] T b20b52a040fb45a0a32100756afe7a81 P 6e3aa0a8326e4168add03993b7255812 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:48:53.204222 2019 raft_consensus.cc:729] T b20b52a040fb45a0a32100756afe7a81 P 6e3aa0a8326e4168add03993b7255812 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e3aa0a8326e4168add03993b7255812, State: Initialized, Role: FOLLOWER
I20260926 20:48:53.204339 2019 consensus_queue.cc:260] T b20b52a040fb45a0a32100756afe7a81 P 6e3aa0a8326e4168add03993b7255812 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d2eebba7c14a698ed9ba4d04bb50d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45635 } } peers { permanent_uuid: "6e3aa0a8326e4168add03993b7255812" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44837 } } peers { permanent_uuid: "0f844a7566be4416bc8dc3f9cb375536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43301 } }
I20260926 20:48:53.204622 2019 ts_tablet_manager.cc:1428] T b20b52a040fb45a0a32100756afe7a81 P 6e3aa0a8326e4168add03993b7255812: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:48:53.204715 1854 heartbeater.cc:503] Master 127.0.98.62:41023 was elected leader, sending a full tablet report...
W20260926 20:48:53.244791 1855 tablet.cc:2369] T b20b52a040fb45a0a32100756afe7a81 P 6e3aa0a8326e4168add03993b7255812: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:48:53.326256 1715 tablet.cc:2369] T b20b52a040fb45a0a32100756afe7a81 P 0f844a7566be4416bc8dc3f9cb375536: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:48:53.360471 2023 raft_consensus.cc:482] T b20b52a040fb45a0a32100756afe7a81 P 0f844a7566be4416bc8dc3f9cb375536 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:48:53.360591 2023 raft_consensus.cc:504] T b20b52a040fb45a0a32100756afe7a81 P 0f844a7566be4416bc8dc3f9cb375536 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d2eebba7c14a698ed9ba4d04bb50d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45635 } } peers { permanent_uuid: "6e3aa0a8326e4168add03993b7255812" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44837 } } peers { permanent_uuid: "0f844a7566be4416bc8dc3f9cb375536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43301 } }
I20260926 20:48:53.360900 2023 leader_election.cc:288] T b20b52a040fb45a0a32100756afe7a81 P 0f844a7566be4416bc8dc3f9cb375536 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1d2eebba7c14a698ed9ba4d04bb50d3 (127.0.98.3:45635), 6e3aa0a8326e4168add03993b7255812 (127.0.98.2:44837)
I20260926 20:48:53.364352 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:53.361303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36701 (local address 127.0.98.3:45635)
0926 20:48:53.361344 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:53.361346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:53.361410 (+ 64us) server_negotiation.cc:407] Connection header received
0926 20:48:53.361439 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:53.361440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:53.361452 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:53.361510 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:48:53.361723 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.362514 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.363544 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.363982 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.364072 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:53.364083 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:53.364249 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 20:48:53.364282 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 20:48:53.364375 2026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:53.361120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45635 (local address 127.0.98.1:36701)
0926 20:48:53.361267 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:53.361280 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:53.361339 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:53.361540 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:53.361541 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:53.361544 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:48:53.361642 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:53.361643 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.362586 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.362587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:53.363444 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:53.363447 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.364075 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.364076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:53.364148 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:53.364153 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:48:53.364242 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 20:48:53.364281 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":25,"thread_start_us":67,"threads_started":1}
I20260926 20:48:53.364460 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:53.361223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42401 (local address 127.0.98.2:44837)
0926 20:48:53.361273 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:53.361275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:53.361340 (+ 65us) server_negotiation.cc:407] Connection header received
0926 20:48:53.361441 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:53.361442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:53.361452 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:53.361510 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:48:53.361723 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.362455 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.363597 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.364088 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.364172 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:53.364182 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:53.364349 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 20:48:53.364385 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 20:48:53.364468 2027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:53.361120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44837 (local address 127.0.98.1:42401)
0926 20:48:53.361267 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:53.361280 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:53.361366 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:53.361540 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:53.361541 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:53.361544 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:48:53.361640 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:53.361642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.362578 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.362579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:53.363510 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:53.363513 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.364184 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.364185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:53.364260 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:53.364263 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:48:53.364340 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 20:48:53.364383 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":24,"spinlock_wait_cycles":15744,"thread_start_us":90,"threads_started":1}
I20260926 20:48:53.364917 1948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b20b52a040fb45a0a32100756afe7a81" candidate_uuid: "0f844a7566be4416bc8dc3f9cb375536" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1d2eebba7c14a698ed9ba4d04bb50d3" is_pre_election: true
I20260926 20:48:53.364921 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b20b52a040fb45a0a32100756afe7a81" candidate_uuid: "0f844a7566be4416bc8dc3f9cb375536" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e3aa0a8326e4168add03993b7255812" is_pre_election: true
I20260926 20:48:53.365070 1948 raft_consensus.cc:2454] T b20b52a040fb45a0a32100756afe7a81 P c1d2eebba7c14a698ed9ba4d04bb50d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f844a7566be4416bc8dc3f9cb375536 in term 0.
I20260926 20:48:53.365077 1808 raft_consensus.cc:2454] T b20b52a040fb45a0a32100756afe7a81 P 6e3aa0a8326e4168add03993b7255812 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f844a7566be4416bc8dc3f9cb375536 in term 0.
I20260926 20:48:53.365289 1601 leader_election.cc:302] T b20b52a040fb45a0a32100756afe7a81 P 0f844a7566be4416bc8dc3f9cb375536 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f844a7566be4416bc8dc3f9cb375536, 6e3aa0a8326e4168add03993b7255812; no voters:
I20260926 20:48:53.365433 2023 raft_consensus.cc:2781] T b20b52a040fb45a0a32100756afe7a81 P 0f844a7566be4416bc8dc3f9cb375536 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:48:53.365518 2023 raft_consensus.cc:482] T b20b52a040fb45a0a32100756afe7a81 P 0f844a7566be4416bc8dc3f9cb375536 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:48:53.365553 2023 raft_consensus.cc:3037] T b20b52a040fb45a0a32100756afe7a81 P 0f844a7566be4416bc8dc3f9cb375536 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:48:53.366298 2023 raft_consensus.cc:504] T b20b52a040fb45a0a32100756afe7a81 P 0f844a7566be4416bc8dc3f9cb375536 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d2eebba7c14a698ed9ba4d04bb50d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45635 } } peers { permanent_uuid: "6e3aa0a8326e4168add03993b7255812" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44837 } } peers { permanent_uuid: "0f844a7566be4416bc8dc3f9cb375536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43301 } }
I20260926 20:48:53.366456 2023 leader_election.cc:288] T b20b52a040fb45a0a32100756afe7a81 P 0f844a7566be4416bc8dc3f9cb375536 [CANDIDATE]: Term 1 election: Requested vote from peers c1d2eebba7c14a698ed9ba4d04bb50d3 (127.0.98.3:45635), 6e3aa0a8326e4168add03993b7255812 (127.0.98.2:44837)
I20260926 20:48:53.366643 1948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b20b52a040fb45a0a32100756afe7a81" candidate_uuid: "0f844a7566be4416bc8dc3f9cb375536" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1d2eebba7c14a698ed9ba4d04bb50d3"
I20260926 20:48:53.366691 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b20b52a040fb45a0a32100756afe7a81" candidate_uuid: "0f844a7566be4416bc8dc3f9cb375536" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e3aa0a8326e4168add03993b7255812"
I20260926 20:48:53.366730 1948 raft_consensus.cc:3037] T b20b52a040fb45a0a32100756afe7a81 P c1d2eebba7c14a698ed9ba4d04bb50d3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:48:53.366791 1808 raft_consensus.cc:3037] T b20b52a040fb45a0a32100756afe7a81 P 6e3aa0a8326e4168add03993b7255812 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:48:53.367468 1948 raft_consensus.cc:2454] T b20b52a040fb45a0a32100756afe7a81 P c1d2eebba7c14a698ed9ba4d04bb50d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f844a7566be4416bc8dc3f9cb375536 in term 1.
I20260926 20:48:53.367552 1808 raft_consensus.cc:2454] T b20b52a040fb45a0a32100756afe7a81 P 6e3aa0a8326e4168add03993b7255812 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f844a7566be4416bc8dc3f9cb375536 in term 1.
I20260926 20:48:53.367668 1602 leader_election.cc:302] T b20b52a040fb45a0a32100756afe7a81 P 0f844a7566be4416bc8dc3f9cb375536 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f844a7566be4416bc8dc3f9cb375536, c1d2eebba7c14a698ed9ba4d04bb50d3; no voters:
I20260926 20:48:53.367770 2023 raft_consensus.cc:2781] T b20b52a040fb45a0a32100756afe7a81 P 0f844a7566be4416bc8dc3f9cb375536 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:48:53.367887 2023 raft_consensus.cc:686] T b20b52a040fb45a0a32100756afe7a81 P 0f844a7566be4416bc8dc3f9cb375536 [term 1 LEADER]: Becoming Leader. State: Replica: 0f844a7566be4416bc8dc3f9cb375536, State: Running, Role: LEADER
I20260926 20:48:53.367990 2023 consensus_queue.cc:237] T b20b52a040fb45a0a32100756afe7a81 P 0f844a7566be4416bc8dc3f9cb375536 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d2eebba7c14a698ed9ba4d04bb50d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45635 } } peers { permanent_uuid: "6e3aa0a8326e4168add03993b7255812" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44837 } } peers { permanent_uuid: "0f844a7566be4416bc8dc3f9cb375536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43301 } }
I20260926 20:48:53.368829 1511 catalog_manager.cc:5495] T b20b52a040fb45a0a32100756afe7a81 P 0f844a7566be4416bc8dc3f9cb375536 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f844a7566be4416bc8dc3f9cb375536 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "0f844a7566be4416bc8dc3f9cb375536" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d2eebba7c14a698ed9ba4d04bb50d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e3aa0a8326e4168add03993b7255812" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f844a7566be4416bc8dc3f9cb375536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43301 } health_report { overall_health: HEALTHY } } }
I20260926 20:48:53.380241 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:53.377876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36674 (local address 127.0.98.1:43301)
0926 20:48:53.377947 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:53.377948 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:53.377957 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:48:53.378034 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:53.378035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:53.378053 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:53.378109 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:48:53.378221 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.378677 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.379432 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.379600 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.379659 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:53.379888 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:48:53.379978 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:48:53.380037 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:53.380113 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 20:48:53.380130 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 20:48:53.382074 1808 raft_consensus.cc:1261] T b20b52a040fb45a0a32100756afe7a81 P 6e3aa0a8326e4168add03993b7255812 [term 1 FOLLOWER]: Refusing update from remote peer 0f844a7566be4416bc8dc3f9cb375536: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:48:53.382102 1948 raft_consensus.cc:1261] T b20b52a040fb45a0a32100756afe7a81 P c1d2eebba7c14a698ed9ba4d04bb50d3 [term 1 FOLLOWER]: Refusing update from remote peer 0f844a7566be4416bc8dc3f9cb375536: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:48:53.382479 2028 consensus_queue.cc:1035] T b20b52a040fb45a0a32100756afe7a81 P 0f844a7566be4416bc8dc3f9cb375536 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1d2eebba7c14a698ed9ba4d04bb50d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:48:53.382622 2023 consensus_queue.cc:1035] T b20b52a040fb45a0a32100756afe7a81 P 0f844a7566be4416bc8dc3f9cb375536 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e3aa0a8326e4168add03993b7255812" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:48:53.384758 2031 mvcc.cc:205] Tried to move back new op lower bound from 7333706683930128384 to 7333706683875893248. Current Snapshot: MvccSnapshot[applied={T|T < 7333706683930128384}]
I20260926 20:48:53.385190 2032 mvcc.cc:205] Tried to move back new op lower bound from 7333706683930128384 to 7333706683875893248. Current Snapshot: MvccSnapshot[applied={T|T < 7333706683930128384}]
I20260926 20:48:53.385404 2033 mvcc.cc:205] Tried to move back new op lower bound from 7333706683930128384 to 7333706683875893248. Current Snapshot: MvccSnapshot[applied={T|T < 7333706683930128384}]
I20260926 20:48:53.388619 1510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:48:53.388724 1510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:48:53.389532 1510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6eb902a2976241ba9a38e69de9023356: Sending DeleteTablet for 3 replicas of tablet b20b52a040fb45a0a32100756afe7a81
I20260926 20:48:53.389889 1928 tablet_service.cc:1514] Processing DeleteTablet for tablet b20b52a040fb45a0a32100756afe7a81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:48:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50854
I20260926 20:48:53.390043 1648 tablet_service.cc:1514] Processing DeleteTablet for tablet b20b52a040fb45a0a32100756afe7a81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:48:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36666
I20260926 20:48:53.390039 1788 tablet_service.cc:1514] Processing DeleteTablet for tablet b20b52a040fb45a0a32100756afe7a81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:48:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57546
I20260926 20:48:53.390465 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48166:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:48:53.390604 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:48:53.391875 1648 tablet_service.cc:1467] Processing CreateTablet for tablet a577cc2345ee48fe805f5df70f82cc3c (DEFAULT_TABLE table=test-table [id=3d74af66b25442708b77cb5468246e07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:48:53.391938 1928 tablet_service.cc:1467] Processing CreateTablet for tablet a577cc2345ee48fe805f5df70f82cc3c (DEFAULT_TABLE table=test-table [id=3d74af66b25442708b77cb5468246e07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:48:53.392001 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a577cc2345ee48fe805f5df70f82cc3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:48:53.392062 1928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a577cc2345ee48fe805f5df70f82cc3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:48:53.393311 2017 tablet_bootstrap.cc:492] T a577cc2345ee48fe805f5df70f82cc3c P 0f844a7566be4416bc8dc3f9cb375536: Bootstrap starting.
I20260926 20:48:53.393712 2018 tablet_bootstrap.cc:492] T a577cc2345ee48fe805f5df70f82cc3c P c1d2eebba7c14a698ed9ba4d04bb50d3: Bootstrap starting.
I20260926 20:48:53.393738 2017 tablet_bootstrap.cc:654] T a577cc2345ee48fe805f5df70f82cc3c P 0f844a7566be4416bc8dc3f9cb375536: Neither blocks nor log segments found. Creating new log.
I20260926 20:48:53.394074 1788 tablet_service.cc:1467] Processing CreateTablet for tablet a577cc2345ee48fe805f5df70f82cc3c (DEFAULT_TABLE table=test-table [id=3d74af66b25442708b77cb5468246e07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:48:53.394134 2018 tablet_bootstrap.cc:654] T a577cc2345ee48fe805f5df70f82cc3c P c1d2eebba7c14a698ed9ba4d04bb50d3: Neither blocks nor log segments found. Creating new log.
I20260926 20:48:53.394213 1788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a577cc2345ee48fe805f5df70f82cc3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:48:53.395701 2019 tablet_bootstrap.cc:492] T a577cc2345ee48fe805f5df70f82cc3c P 6e3aa0a8326e4168add03993b7255812: Bootstrap starting.
I20260926 20:48:53.396130 2019 tablet_bootstrap.cc:654] T a577cc2345ee48fe805f5df70f82cc3c P 6e3aa0a8326e4168add03993b7255812: Neither blocks nor log segments found. Creating new log.
I20260926 20:48:53.396829 2019 tablet_bootstrap.cc:492] T a577cc2345ee48fe805f5df70f82cc3c P 6e3aa0a8326e4168add03993b7255812: No bootstrap required, opened a new log
I20260926 20:48:53.396884 2019 ts_tablet_manager.cc:1397] T a577cc2345ee48fe805f5df70f82cc3c P 6e3aa0a8326e4168add03993b7255812: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:48:53.397032 2019 raft_consensus.cc:348] T a577cc2345ee48fe805f5df70f82cc3c P 6e3aa0a8326e4168add03993b7255812 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f844a7566be4416bc8dc3f9cb375536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43301 } } peers { permanent_uuid: "6e3aa0a8326e4168add03993b7255812" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44837 } } peers { permanent_uuid: "c1d2eebba7c14a698ed9ba4d04bb50d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45635 } }
I20260926 20:48:53.397114 2019 raft_consensus.cc:374] T a577cc2345ee48fe805f5df70f82cc3c P 6e3aa0a8326e4168add03993b7255812 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:48:53.397140 2019 raft_consensus.cc:729] T a577cc2345ee48fe805f5df70f82cc3c P 6e3aa0a8326e4168add03993b7255812 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e3aa0a8326e4168add03993b7255812, State: Initialized, Role: FOLLOWER
I20260926 20:48:53.397187 2019 consensus_queue.cc:260] T a577cc2345ee48fe805f5df70f82cc3c P 6e3aa0a8326e4168add03993b7255812 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f844a7566be4416bc8dc3f9cb375536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43301 } } peers { permanent_uuid: "6e3aa0a8326e4168add03993b7255812" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44837 } } peers { permanent_uuid: "c1d2eebba7c14a698ed9ba4d04bb50d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45635 } }
I20260926 20:48:53.397656 2041 tablet_replica.cc:330] T b20b52a040fb45a0a32100756afe7a81 P c1d2eebba7c14a698ed9ba4d04bb50d3: stopping tablet replica
I20260926 20:48:53.397681 2019 ts_tablet_manager.cc:1428] T a577cc2345ee48fe805f5df70f82cc3c P 6e3aa0a8326e4168add03993b7255812: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:48:53.397781 2041 raft_consensus.cc:2229] T b20b52a040fb45a0a32100756afe7a81 P c1d2eebba7c14a698ed9ba4d04bb50d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:48:53.397948 2041 raft_consensus.cc:2258] T b20b52a040fb45a0a32100756afe7a81 P c1d2eebba7c14a698ed9ba4d04bb50d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:48:53.398120 2042 tablet_replica.cc:330] T b20b52a040fb45a0a32100756afe7a81 P 0f844a7566be4416bc8dc3f9cb375536: stopping tablet replica
I20260926 20:48:53.398276 2043 tablet_replica.cc:330] T b20b52a040fb45a0a32100756afe7a81 P 6e3aa0a8326e4168add03993b7255812: stopping tablet replica
I20260926 20:48:53.398382 2043 raft_consensus.cc:2229] T b20b52a040fb45a0a32100756afe7a81 P 6e3aa0a8326e4168add03993b7255812 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:48:53.398481 2043 raft_consensus.cc:2258] T b20b52a040fb45a0a32100756afe7a81 P 6e3aa0a8326e4168add03993b7255812 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:48:53.398471 2041 ts_tablet_manager.cc:1905] T b20b52a040fb45a0a32100756afe7a81 P c1d2eebba7c14a698ed9ba4d04bb50d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:48:53.398630 2042 raft_consensus.cc:2229] T b20b52a040fb45a0a32100756afe7a81 P 0f844a7566be4416bc8dc3f9cb375536 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:48:53.398797 2042 raft_consensus.cc:2258] T b20b52a040fb45a0a32100756afe7a81 P 0f844a7566be4416bc8dc3f9cb375536 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:48:53.399238 2042 ts_tablet_manager.cc:1905] T b20b52a040fb45a0a32100756afe7a81 P 0f844a7566be4416bc8dc3f9cb375536: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:48:53.400283 2041 ts_tablet_manager.cc:1918] T b20b52a040fb45a0a32100756afe7a81 P c1d2eebba7c14a698ed9ba4d04bb50d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:48:53.400357 2041 log.cc:1196] T b20b52a040fb45a0a32100756afe7a81 P c1d2eebba7c14a698ed9ba4d04bb50d3: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-2/wal/wals/b20b52a040fb45a0a32100756afe7a81
I20260926 20:48:53.400605 2041 ts_tablet_manager.cc:1939] T b20b52a040fb45a0a32100756afe7a81 P c1d2eebba7c14a698ed9ba4d04bb50d3: Deleting consensus metadata
I20260926 20:48:53.400835 2042 ts_tablet_manager.cc:1918] T b20b52a040fb45a0a32100756afe7a81 P 0f844a7566be4416bc8dc3f9cb375536: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:48:53.400921 2042 log.cc:1196] T b20b52a040fb45a0a32100756afe7a81 P 0f844a7566be4416bc8dc3f9cb375536: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-0/wal/wals/b20b52a040fb45a0a32100756afe7a81
I20260926 20:48:53.401089 1498 catalog_manager.cc:4841] TS c1d2eebba7c14a698ed9ba4d04bb50d3 (127.0.98.3:45635): tablet b20b52a040fb45a0a32100756afe7a81 (table test-table [id=457671e304ca4390a5389f77d7c0a49d]) successfully deleted
I20260926 20:48:53.401255 2042 ts_tablet_manager.cc:1939] T b20b52a040fb45a0a32100756afe7a81 P 0f844a7566be4416bc8dc3f9cb375536: Deleting consensus metadata
I20260926 20:48:53.401652 1496 catalog_manager.cc:4841] TS 0f844a7566be4416bc8dc3f9cb375536 (127.0.98.1:43301): tablet b20b52a040fb45a0a32100756afe7a81 (table test-table [id=457671e304ca4390a5389f77d7c0a49d]) successfully deleted
I20260926 20:48:53.403998 2043 ts_tablet_manager.cc:1905] T b20b52a040fb45a0a32100756afe7a81 P 6e3aa0a8326e4168add03993b7255812: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:48:53.404170 2017 tablet_bootstrap.cc:492] T a577cc2345ee48fe805f5df70f82cc3c P 0f844a7566be4416bc8dc3f9cb375536: No bootstrap required, opened a new log
I20260926 20:48:53.404223 2017 ts_tablet_manager.cc:1397] T a577cc2345ee48fe805f5df70f82cc3c P 0f844a7566be4416bc8dc3f9cb375536: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20260926 20:48:53.404338 2018 tablet_bootstrap.cc:492] T a577cc2345ee48fe805f5df70f82cc3c P c1d2eebba7c14a698ed9ba4d04bb50d3: No bootstrap required, opened a new log
I20260926 20:48:53.404395 2018 ts_tablet_manager.cc:1397] T a577cc2345ee48fe805f5df70f82cc3c P c1d2eebba7c14a698ed9ba4d04bb50d3: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.001s
I20260926 20:48:53.404402 2017 raft_consensus.cc:348] T a577cc2345ee48fe805f5df70f82cc3c P 0f844a7566be4416bc8dc3f9cb375536 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f844a7566be4416bc8dc3f9cb375536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43301 } } peers { permanent_uuid: "6e3aa0a8326e4168add03993b7255812" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44837 } } peers { permanent_uuid: "c1d2eebba7c14a698ed9ba4d04bb50d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45635 } }
I20260926 20:48:53.404491 2017 raft_consensus.cc:374] T a577cc2345ee48fe805f5df70f82cc3c P 0f844a7566be4416bc8dc3f9cb375536 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:48:53.404520 2017 raft_consensus.cc:729] T a577cc2345ee48fe805f5df70f82cc3c P 0f844a7566be4416bc8dc3f9cb375536 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f844a7566be4416bc8dc3f9cb375536, State: Initialized, Role: FOLLOWER
I20260926 20:48:53.404570 2017 consensus_queue.cc:260] T a577cc2345ee48fe805f5df70f82cc3c P 0f844a7566be4416bc8dc3f9cb375536 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f844a7566be4416bc8dc3f9cb375536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43301 } } peers { permanent_uuid: "6e3aa0a8326e4168add03993b7255812" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44837 } } peers { permanent_uuid: "c1d2eebba7c14a698ed9ba4d04bb50d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45635 } }
I20260926 20:48:53.404536 2018 raft_consensus.cc:348] T a577cc2345ee48fe805f5df70f82cc3c P c1d2eebba7c14a698ed9ba4d04bb50d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f844a7566be4416bc8dc3f9cb375536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43301 } } peers { permanent_uuid: "6e3aa0a8326e4168add03993b7255812" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44837 } } peers { permanent_uuid: "c1d2eebba7c14a698ed9ba4d04bb50d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45635 } }
I20260926 20:48:53.404654 2018 raft_consensus.cc:374] T a577cc2345ee48fe805f5df70f82cc3c P c1d2eebba7c14a698ed9ba4d04bb50d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:48:53.404675 2017 ts_tablet_manager.cc:1428] T a577cc2345ee48fe805f5df70f82cc3c P 0f844a7566be4416bc8dc3f9cb375536: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 20:48:53.404682 2018 raft_consensus.cc:729] T a577cc2345ee48fe805f5df70f82cc3c P c1d2eebba7c14a698ed9ba4d04bb50d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1d2eebba7c14a698ed9ba4d04bb50d3, State: Initialized, Role: FOLLOWER
I20260926 20:48:53.404732 2018 consensus_queue.cc:260] T a577cc2345ee48fe805f5df70f82cc3c P c1d2eebba7c14a698ed9ba4d04bb50d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f844a7566be4416bc8dc3f9cb375536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43301 } } peers { permanent_uuid: "6e3aa0a8326e4168add03993b7255812" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44837 } } peers { permanent_uuid: "c1d2eebba7c14a698ed9ba4d04bb50d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45635 } }
I20260926 20:48:53.404824 2018 ts_tablet_manager.cc:1428] T a577cc2345ee48fe805f5df70f82cc3c P c1d2eebba7c14a698ed9ba4d04bb50d3: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 20:48:53.405735 2043 ts_tablet_manager.cc:1918] T b20b52a040fb45a0a32100756afe7a81 P 6e3aa0a8326e4168add03993b7255812: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:48:53.405807 2043 log.cc:1196] T b20b52a040fb45a0a32100756afe7a81 P 6e3aa0a8326e4168add03993b7255812: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-1/wal/wals/b20b52a040fb45a0a32100756afe7a81
I20260926 20:48:53.406068 2043 ts_tablet_manager.cc:1939] T b20b52a040fb45a0a32100756afe7a81 P 6e3aa0a8326e4168add03993b7255812: Deleting consensus metadata
I20260926 20:48:53.406477 1497 catalog_manager.cc:4841] TS 6e3aa0a8326e4168add03993b7255812 (127.0.98.2:44837): tablet b20b52a040fb45a0a32100756afe7a81 (table test-table [id=457671e304ca4390a5389f77d7c0a49d]) successfully deleted
I20260926 20:48:53.543337 2025 raft_consensus.cc:482] T a577cc2345ee48fe805f5df70f82cc3c P 6e3aa0a8326e4168add03993b7255812 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:48:53.543470 2025 raft_consensus.cc:504] T a577cc2345ee48fe805f5df70f82cc3c P 6e3aa0a8326e4168add03993b7255812 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f844a7566be4416bc8dc3f9cb375536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43301 } } peers { permanent_uuid: "6e3aa0a8326e4168add03993b7255812" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44837 } } peers { permanent_uuid: "c1d2eebba7c14a698ed9ba4d04bb50d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45635 } }
I20260926 20:48:53.543948 2025 leader_election.cc:288] T a577cc2345ee48fe805f5df70f82cc3c P 6e3aa0a8326e4168add03993b7255812 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f844a7566be4416bc8dc3f9cb375536 (127.0.98.1:43301), c1d2eebba7c14a698ed9ba4d04bb50d3 (127.0.98.3:45635)
I20260926 20:48:53.547501 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:53.544274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35709 (local address 127.0.98.1:43301)
0926 20:48:53.544333 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:53.544335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:53.544346 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:48:53.544445 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:53.544447 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:53.544456 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:53.544490 (+ 34us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:48:53.544771 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.545635 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.546828 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.547193 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.547250 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:53.547258 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:53.547431 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 20:48:53.547458 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 20:48:53.547547 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:53.544137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43301 (local address 127.0.98.2:35709)
0926 20:48:53.544263 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:53.544275 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:53.544345 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:53.544589 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:53.544590 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:53.544594 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:48:53.544687 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:53.544689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.545735 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.545736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:53.546740 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:53.546742 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.547283 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.547284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:53.547352 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:53.547356 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:48:53.547416 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 20:48:53.547455 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":26,"thread_start_us":62,"threads_started":1}
I20260926 20:48:53.547827 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a577cc2345ee48fe805f5df70f82cc3c" candidate_uuid: "6e3aa0a8326e4168add03993b7255812" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f844a7566be4416bc8dc3f9cb375536" is_pre_election: true
I20260926 20:48:53.547957 1668 raft_consensus.cc:2454] T a577cc2345ee48fe805f5df70f82cc3c P 0f844a7566be4416bc8dc3f9cb375536 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e3aa0a8326e4168add03993b7255812 in term 0.
I20260926 20:48:53.548174 1740 leader_election.cc:302] T a577cc2345ee48fe805f5df70f82cc3c P 6e3aa0a8326e4168add03993b7255812 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f844a7566be4416bc8dc3f9cb375536, 6e3aa0a8326e4168add03993b7255812; no voters:
I20260926 20:48:53.548377 2025 raft_consensus.cc:2781] T a577cc2345ee48fe805f5df70f82cc3c P 6e3aa0a8326e4168add03993b7255812 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:48:53.548441 2025 raft_consensus.cc:482] T a577cc2345ee48fe805f5df70f82cc3c P 6e3aa0a8326e4168add03993b7255812 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:48:53.548475 2025 raft_consensus.cc:3037] T a577cc2345ee48fe805f5df70f82cc3c P 6e3aa0a8326e4168add03993b7255812 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:48:53.548637 2047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:53.544137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45635 (local address 127.0.98.2:32961)
0926 20:48:53.544327 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:53.544336 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:53.544377 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:53.544589 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:53.544590 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:53.544592 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:48:53.544686 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:53.544689 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.546651 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.546652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:53.547530 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:53.547532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.548381 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.548382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:53.548438 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:53.548442 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:48:53.548517 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 20:48:53.548565 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":2,"thread_start_us":59,"threads_started":1}
I20260926 20:48:53.548678 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:53.544387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32961 (local address 127.0.98.3:45635)
0926 20:48:53.544431 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:53.544433 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:53.544442 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:48:53.544470 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:53.544472 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:53.544482 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:53.544562 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:48:53.545750 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.546545 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.547658 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.548262 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.548334 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:53.548353 (+ 19us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:53.548554 (+ 201us) server_negotiation.cc:299] Negotiation successful
0926 20:48:53.548604 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 20:48:53.549232 2025 raft_consensus.cc:504] T a577cc2345ee48fe805f5df70f82cc3c P 6e3aa0a8326e4168add03993b7255812 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f844a7566be4416bc8dc3f9cb375536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43301 } } peers { permanent_uuid: "6e3aa0a8326e4168add03993b7255812" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44837 } } peers { permanent_uuid: "c1d2eebba7c14a698ed9ba4d04bb50d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45635 } }
I20260926 20:48:53.549417 2025 leader_election.cc:288] T a577cc2345ee48fe805f5df70f82cc3c P 6e3aa0a8326e4168add03993b7255812 [CANDIDATE]: Term 1 election: Requested vote from peers 0f844a7566be4416bc8dc3f9cb375536 (127.0.98.1:43301), c1d2eebba7c14a698ed9ba4d04bb50d3 (127.0.98.3:45635)
I20260926 20:48:53.549511 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a577cc2345ee48fe805f5df70f82cc3c" candidate_uuid: "6e3aa0a8326e4168add03993b7255812" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f844a7566be4416bc8dc3f9cb375536"
I20260926 20:48:53.549605 1668 raft_consensus.cc:3037] T a577cc2345ee48fe805f5df70f82cc3c P 0f844a7566be4416bc8dc3f9cb375536 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:48:53.549630 1948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a577cc2345ee48fe805f5df70f82cc3c" candidate_uuid: "6e3aa0a8326e4168add03993b7255812" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1d2eebba7c14a698ed9ba4d04bb50d3" is_pre_election: true
I20260926 20:48:53.549667 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a577cc2345ee48fe805f5df70f82cc3c" candidate_uuid: "6e3aa0a8326e4168add03993b7255812" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1d2eebba7c14a698ed9ba4d04bb50d3"
I20260926 20:48:53.549710 1948 raft_consensus.cc:2454] T a577cc2345ee48fe805f5df70f82cc3c P c1d2eebba7c14a698ed9ba4d04bb50d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e3aa0a8326e4168add03993b7255812 in term 0.
I20260926 20:48:53.550302 1668 raft_consensus.cc:2454] T a577cc2345ee48fe805f5df70f82cc3c P 0f844a7566be4416bc8dc3f9cb375536 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e3aa0a8326e4168add03993b7255812 in term 1.
I20260926 20:48:53.550522 1740 leader_election.cc:302] T a577cc2345ee48fe805f5df70f82cc3c P 6e3aa0a8326e4168add03993b7255812 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0f844a7566be4416bc8dc3f9cb375536, 6e3aa0a8326e4168add03993b7255812; no voters: c1d2eebba7c14a698ed9ba4d04bb50d3
I20260926 20:48:53.550642 2025 raft_consensus.cc:2781] T a577cc2345ee48fe805f5df70f82cc3c P 6e3aa0a8326e4168add03993b7255812 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:48:53.550768 2025 raft_consensus.cc:686] T a577cc2345ee48fe805f5df70f82cc3c P 6e3aa0a8326e4168add03993b7255812 [term 1 LEADER]: Becoming Leader. State: Replica: 6e3aa0a8326e4168add03993b7255812, State: Running, Role: LEADER
I20260926 20:48:53.550836 2025 consensus_queue.cc:237] T a577cc2345ee48fe805f5df70f82cc3c P 6e3aa0a8326e4168add03993b7255812 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f844a7566be4416bc8dc3f9cb375536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43301 } } peers { permanent_uuid: "6e3aa0a8326e4168add03993b7255812" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44837 } } peers { permanent_uuid: "c1d2eebba7c14a698ed9ba4d04bb50d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45635 } }
I20260926 20:48:53.551451 1511 catalog_manager.cc:5495] T a577cc2345ee48fe805f5df70f82cc3c P 6e3aa0a8326e4168add03993b7255812 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e3aa0a8326e4168add03993b7255812 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "6e3aa0a8326e4168add03993b7255812" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f844a7566be4416bc8dc3f9cb375536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e3aa0a8326e4168add03993b7255812" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1d2eebba7c14a698ed9ba4d04bb50d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45635 } health_report { overall_health: UNKNOWN } } }
Sep 26 20:48:53 dist-test-slave-hwlj krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455732, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:48:53.611773 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:53.604648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48182 (local address 127.0.98.62:41023)
0926 20:48:53.604717 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:53.604719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:53.604815 (+ 96us) server_negotiation.cc:407] Connection header received
0926 20:48:53.604917 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:53.604918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:53.604928 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:53.605001 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:53.605445 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.605933 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.606693 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.606878 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.606967 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:53.609143 (+ 2176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:53.609144 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:53.609145 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:53.609152 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:53.610514 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:53.610898 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:53.610900 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:53.610900 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:53.610953 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:53.611163 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:53.611164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:53.611164 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:53.611276 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:53.611356 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:53.611520 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 20:48:53.611676 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":88,"server-negotiator.queue_time_us":29}
I20260926 20:48:53.611776 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:53.604558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41023 (local address 127.0.0.1:48182)
0926 20:48:53.604724 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:53.604735 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:53.604833 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:53.605004 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:53.605005 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:53.605309 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:48:53.605354 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:53.605356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.606065 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.606066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:53.606586 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:53.606588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.606988 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.606989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:53.607040 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:53.607577 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:48:53.607581 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:48:53.609039 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:48:53.610603 (+ 1564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:53.610611 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:53.610613 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:53.610807 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:53.611019 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:53.611020 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:53.611020 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:53.611076 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:53.611346 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:48:53.611353 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:48:53.611402 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 20:48:53.611487 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 20:48:53.611624 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":2,"thread_start_us":91,"threads_started":1}
I20260926 20:48:53.614308 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48182:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 20:48:53.614578 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:48:53.616147 1648 tablet_service.cc:1467] Processing CreateTablet for tablet 88412d90d7cb4239af85aff01bfab87f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c41b394d94ca44a58c03fedbb922efc2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:48:53.616120 1788 tablet_service.cc:1467] Processing CreateTablet for tablet 88412d90d7cb4239af85aff01bfab87f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c41b394d94ca44a58c03fedbb922efc2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:48:53.616184 1928 tablet_service.cc:1467] Processing CreateTablet for tablet 88412d90d7cb4239af85aff01bfab87f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c41b394d94ca44a58c03fedbb922efc2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:48:53.616309 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88412d90d7cb4239af85aff01bfab87f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:48:53.616309 1788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88412d90d7cb4239af85aff01bfab87f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:48:53.616348 1928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88412d90d7cb4239af85aff01bfab87f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:48:53.617514 2018 tablet_bootstrap.cc:492] T 88412d90d7cb4239af85aff01bfab87f P c1d2eebba7c14a698ed9ba4d04bb50d3: Bootstrap starting.
I20260926 20:48:53.617786 2019 tablet_bootstrap.cc:492] T 88412d90d7cb4239af85aff01bfab87f P 6e3aa0a8326e4168add03993b7255812: Bootstrap starting.
I20260926 20:48:53.617797 2017 tablet_bootstrap.cc:492] T 88412d90d7cb4239af85aff01bfab87f P 0f844a7566be4416bc8dc3f9cb375536: Bootstrap starting.
I20260926 20:48:53.617962 2018 tablet_bootstrap.cc:654] T 88412d90d7cb4239af85aff01bfab87f P c1d2eebba7c14a698ed9ba4d04bb50d3: Neither blocks nor log segments found. Creating new log.
I20260926 20:48:53.618062 2017 tablet_bootstrap.cc:654] T 88412d90d7cb4239af85aff01bfab87f P 0f844a7566be4416bc8dc3f9cb375536: Neither blocks nor log segments found. Creating new log.
I20260926 20:48:53.618189 2019 tablet_bootstrap.cc:654] T 88412d90d7cb4239af85aff01bfab87f P 6e3aa0a8326e4168add03993b7255812: Neither blocks nor log segments found. Creating new log.
I20260926 20:48:53.618600 2017 tablet_bootstrap.cc:492] T 88412d90d7cb4239af85aff01bfab87f P 0f844a7566be4416bc8dc3f9cb375536: No bootstrap required, opened a new log
I20260926 20:48:53.618676 2017 ts_tablet_manager.cc:1397] T 88412d90d7cb4239af85aff01bfab87f P 0f844a7566be4416bc8dc3f9cb375536: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:48:53.618839 2019 tablet_bootstrap.cc:492] T 88412d90d7cb4239af85aff01bfab87f P 6e3aa0a8326e4168add03993b7255812: No bootstrap required, opened a new log
I20260926 20:48:53.618805 2017 raft_consensus.cc:348] T 88412d90d7cb4239af85aff01bfab87f P 0f844a7566be4416bc8dc3f9cb375536 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f844a7566be4416bc8dc3f9cb375536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43301 } } peers { permanent_uuid: "6e3aa0a8326e4168add03993b7255812" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44837 } } peers { permanent_uuid: "c1d2eebba7c14a698ed9ba4d04bb50d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45635 } }
I20260926 20:48:53.618877 2017 raft_consensus.cc:374] T 88412d90d7cb4239af85aff01bfab87f P 0f844a7566be4416bc8dc3f9cb375536 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:48:53.618885 2019 ts_tablet_manager.cc:1397] T 88412d90d7cb4239af85aff01bfab87f P 6e3aa0a8326e4168add03993b7255812: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:48:53.618912 2017 raft_consensus.cc:729] T 88412d90d7cb4239af85aff01bfab87f P 0f844a7566be4416bc8dc3f9cb375536 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f844a7566be4416bc8dc3f9cb375536, State: Initialized, Role: FOLLOWER
I20260926 20:48:53.618974 2017 consensus_queue.cc:260] T 88412d90d7cb4239af85aff01bfab87f P 0f844a7566be4416bc8dc3f9cb375536 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f844a7566be4416bc8dc3f9cb375536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43301 } } peers { permanent_uuid: "6e3aa0a8326e4168add03993b7255812" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44837 } } peers { permanent_uuid: "c1d2eebba7c14a698ed9ba4d04bb50d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45635 } }
I20260926 20:48:53.619027 2019 raft_consensus.cc:348] T 88412d90d7cb4239af85aff01bfab87f P 6e3aa0a8326e4168add03993b7255812 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f844a7566be4416bc8dc3f9cb375536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43301 } } peers { permanent_uuid: "6e3aa0a8326e4168add03993b7255812" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44837 } } peers { permanent_uuid: "c1d2eebba7c14a698ed9ba4d04bb50d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45635 } }
I20260926 20:48:53.619091 2017 ts_tablet_manager.cc:1428] T 88412d90d7cb4239af85aff01bfab87f P 0f844a7566be4416bc8dc3f9cb375536: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:48:53.619107 2019 raft_consensus.cc:374] T 88412d90d7cb4239af85aff01bfab87f P 6e3aa0a8326e4168add03993b7255812 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:48:53.619136 2019 raft_consensus.cc:729] T 88412d90d7cb4239af85aff01bfab87f P 6e3aa0a8326e4168add03993b7255812 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e3aa0a8326e4168add03993b7255812, State: Initialized, Role: FOLLOWER
I20260926 20:48:53.619120 2036 tablet_replica.cc:439] T 88412d90d7cb4239af85aff01bfab87f P 0f844a7566be4416bc8dc3f9cb375536: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f844a7566be4416bc8dc3f9cb375536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43301 } } peers { permanent_uuid: "6e3aa0a8326e4168add03993b7255812" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44837 } } peers { permanent_uuid: "c1d2eebba7c14a698ed9ba4d04bb50d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45635 } } }
I20260926 20:48:53.619182 2019 consensus_queue.cc:260] T 88412d90d7cb4239af85aff01bfab87f P 6e3aa0a8326e4168add03993b7255812 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f844a7566be4416bc8dc3f9cb375536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43301 } } peers { permanent_uuid: "6e3aa0a8326e4168add03993b7255812" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44837 } } peers { permanent_uuid: "c1d2eebba7c14a698ed9ba4d04bb50d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45635 } }
I20260926 20:48:53.619227 2036 tablet_replica.cc:442] T 88412d90d7cb4239af85aff01bfab87f P 0f844a7566be4416bc8dc3f9cb375536: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:48:53.619263 2019 ts_tablet_manager.cc:1428] T 88412d90d7cb4239af85aff01bfab87f P 6e3aa0a8326e4168add03993b7255812: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:48:53.619375 2025 tablet_replica.cc:439] T 88412d90d7cb4239af85aff01bfab87f P 6e3aa0a8326e4168add03993b7255812: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f844a7566be4416bc8dc3f9cb375536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43301 } } peers { permanent_uuid: "6e3aa0a8326e4168add03993b7255812" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44837 } } peers { permanent_uuid: "c1d2eebba7c14a698ed9ba4d04bb50d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45635 } } }
I20260926 20:48:53.619474 2025 tablet_replica.cc:442] T 88412d90d7cb4239af85aff01bfab87f P 6e3aa0a8326e4168add03993b7255812: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:48:53.619774 2018 tablet_bootstrap.cc:492] T 88412d90d7cb4239af85aff01bfab87f P c1d2eebba7c14a698ed9ba4d04bb50d3: No bootstrap required, opened a new log
I20260926 20:48:53.619827 2018 ts_tablet_manager.cc:1397] T 88412d90d7cb4239af85aff01bfab87f P c1d2eebba7c14a698ed9ba4d04bb50d3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 20:48:53.619951 2018 raft_consensus.cc:348] T 88412d90d7cb4239af85aff01bfab87f P c1d2eebba7c14a698ed9ba4d04bb50d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f844a7566be4416bc8dc3f9cb375536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43301 } } peers { permanent_uuid: "6e3aa0a8326e4168add03993b7255812" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44837 } } peers { permanent_uuid: "c1d2eebba7c14a698ed9ba4d04bb50d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45635 } }
I20260926 20:48:53.620003 2018 raft_consensus.cc:374] T 88412d90d7cb4239af85aff01bfab87f P c1d2eebba7c14a698ed9ba4d04bb50d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:48:53.620064 2018 raft_consensus.cc:729] T 88412d90d7cb4239af85aff01bfab87f P c1d2eebba7c14a698ed9ba4d04bb50d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1d2eebba7c14a698ed9ba4d04bb50d3, State: Initialized, Role: FOLLOWER
I20260926 20:48:53.620118 2018 consensus_queue.cc:260] T 88412d90d7cb4239af85aff01bfab87f P c1d2eebba7c14a698ed9ba4d04bb50d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f844a7566be4416bc8dc3f9cb375536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43301 } } peers { permanent_uuid: "6e3aa0a8326e4168add03993b7255812" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44837 } } peers { permanent_uuid: "c1d2eebba7c14a698ed9ba4d04bb50d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45635 } }
I20260926 20:48:53.620219 2018 ts_tablet_manager.cc:1428] T 88412d90d7cb4239af85aff01bfab87f P c1d2eebba7c14a698ed9ba4d04bb50d3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:48:53.620368 2024 tablet_replica.cc:439] T 88412d90d7cb4239af85aff01bfab87f P c1d2eebba7c14a698ed9ba4d04bb50d3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f844a7566be4416bc8dc3f9cb375536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43301 } } peers { permanent_uuid: "6e3aa0a8326e4168add03993b7255812" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44837 } } peers { permanent_uuid: "c1d2eebba7c14a698ed9ba4d04bb50d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45635 } } }
I20260926 20:48:53.620452 2024 tablet_replica.cc:442] T 88412d90d7cb4239af85aff01bfab87f P c1d2eebba7c14a698ed9ba4d04bb50d3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:48:53.636613 2025 raft_consensus.cc:482] T 88412d90d7cb4239af85aff01bfab87f P 6e3aa0a8326e4168add03993b7255812 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:48:53.636706 2025 raft_consensus.cc:504] T 88412d90d7cb4239af85aff01bfab87f P 6e3aa0a8326e4168add03993b7255812 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f844a7566be4416bc8dc3f9cb375536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43301 } } peers { permanent_uuid: "6e3aa0a8326e4168add03993b7255812" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44837 } } peers { permanent_uuid: "c1d2eebba7c14a698ed9ba4d04bb50d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45635 } }
I20260926 20:48:53.636878 2025 leader_election.cc:288] T 88412d90d7cb4239af85aff01bfab87f P 6e3aa0a8326e4168add03993b7255812 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f844a7566be4416bc8dc3f9cb375536 (127.0.98.1:43301), c1d2eebba7c14a698ed9ba4d04bb50d3 (127.0.98.3:45635)
I20260926 20:48:53.637074 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88412d90d7cb4239af85aff01bfab87f" candidate_uuid: "6e3aa0a8326e4168add03993b7255812" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f844a7566be4416bc8dc3f9cb375536" is_pre_election: true
I20260926 20:48:53.637116 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88412d90d7cb4239af85aff01bfab87f" candidate_uuid: "6e3aa0a8326e4168add03993b7255812" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1d2eebba7c14a698ed9ba4d04bb50d3" is_pre_election: true
I20260926 20:48:53.637188 1668 raft_consensus.cc:2454] T 88412d90d7cb4239af85aff01bfab87f P 0f844a7566be4416bc8dc3f9cb375536 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e3aa0a8326e4168add03993b7255812 in term 0.
I20260926 20:48:53.637197 1947 raft_consensus.cc:2454] T 88412d90d7cb4239af85aff01bfab87f P c1d2eebba7c14a698ed9ba4d04bb50d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e3aa0a8326e4168add03993b7255812 in term 0.
I20260926 20:48:53.637372 1740 leader_election.cc:302] T 88412d90d7cb4239af85aff01bfab87f P 6e3aa0a8326e4168add03993b7255812 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f844a7566be4416bc8dc3f9cb375536, 6e3aa0a8326e4168add03993b7255812; no voters:
I20260926 20:48:53.637487 2025 raft_consensus.cc:2781] T 88412d90d7cb4239af85aff01bfab87f P 6e3aa0a8326e4168add03993b7255812 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:48:53.637565 2025 raft_consensus.cc:482] T 88412d90d7cb4239af85aff01bfab87f P 6e3aa0a8326e4168add03993b7255812 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:48:53.637599 2025 raft_consensus.cc:3037] T 88412d90d7cb4239af85aff01bfab87f P 6e3aa0a8326e4168add03993b7255812 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:48:53.638715 2025 raft_consensus.cc:504] T 88412d90d7cb4239af85aff01bfab87f P 6e3aa0a8326e4168add03993b7255812 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f844a7566be4416bc8dc3f9cb375536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43301 } } peers { permanent_uuid: "6e3aa0a8326e4168add03993b7255812" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44837 } } peers { permanent_uuid: "c1d2eebba7c14a698ed9ba4d04bb50d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45635 } }
I20260926 20:48:53.638886 2025 leader_election.cc:288] T 88412d90d7cb4239af85aff01bfab87f P 6e3aa0a8326e4168add03993b7255812 [CANDIDATE]: Term 1 election: Requested vote from peers 0f844a7566be4416bc8dc3f9cb375536 (127.0.98.1:43301), c1d2eebba7c14a698ed9ba4d04bb50d3 (127.0.98.3:45635)
I20260926 20:48:53.639040 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88412d90d7cb4239af85aff01bfab87f" candidate_uuid: "6e3aa0a8326e4168add03993b7255812" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1d2eebba7c14a698ed9ba4d04bb50d3"
I20260926 20:48:53.639055 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88412d90d7cb4239af85aff01bfab87f" candidate_uuid: "6e3aa0a8326e4168add03993b7255812" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f844a7566be4416bc8dc3f9cb375536"
I20260926 20:48:53.639153 1668 raft_consensus.cc:3037] T 88412d90d7cb4239af85aff01bfab87f P 0f844a7566be4416bc8dc3f9cb375536 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:48:53.639139 1947 raft_consensus.cc:3037] T 88412d90d7cb4239af85aff01bfab87f P c1d2eebba7c14a698ed9ba4d04bb50d3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:48:53.639818 1668 raft_consensus.cc:2454] T 88412d90d7cb4239af85aff01bfab87f P 0f844a7566be4416bc8dc3f9cb375536 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e3aa0a8326e4168add03993b7255812 in term 1.
I20260926 20:48:53.639906 1947 raft_consensus.cc:2454] T 88412d90d7cb4239af85aff01bfab87f P c1d2eebba7c14a698ed9ba4d04bb50d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e3aa0a8326e4168add03993b7255812 in term 1.
I20260926 20:48:53.639981 1740 leader_election.cc:302] T 88412d90d7cb4239af85aff01bfab87f P 6e3aa0a8326e4168add03993b7255812 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f844a7566be4416bc8dc3f9cb375536, 6e3aa0a8326e4168add03993b7255812; no voters:
I20260926 20:48:53.640080 2025 raft_consensus.cc:2781] T 88412d90d7cb4239af85aff01bfab87f P 6e3aa0a8326e4168add03993b7255812 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:48:53.640126 2025 raft_consensus.cc:686] T 88412d90d7cb4239af85aff01bfab87f P 6e3aa0a8326e4168add03993b7255812 [term 1 LEADER]: Becoming Leader. State: Replica: 6e3aa0a8326e4168add03993b7255812, State: Running, Role: LEADER
I20260926 20:48:53.640174 2025 consensus_queue.cc:237] T 88412d90d7cb4239af85aff01bfab87f P 6e3aa0a8326e4168add03993b7255812 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f844a7566be4416bc8dc3f9cb375536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43301 } } peers { permanent_uuid: "6e3aa0a8326e4168add03993b7255812" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44837 } } peers { permanent_uuid: "c1d2eebba7c14a698ed9ba4d04bb50d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45635 } }
I20260926 20:48:53.640515 2049 tablet_replica.cc:439] T 88412d90d7cb4239af85aff01bfab87f P 6e3aa0a8326e4168add03993b7255812: TxnStatusTablet state changed. Reason: New leader 6e3aa0a8326e4168add03993b7255812. Latest consensus state: current_term: 1 leader_uuid: "6e3aa0a8326e4168add03993b7255812" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f844a7566be4416bc8dc3f9cb375536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43301 } } peers { permanent_uuid: "6e3aa0a8326e4168add03993b7255812" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44837 } } peers { permanent_uuid: "c1d2eebba7c14a698ed9ba4d04bb50d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45635 } } }
I20260926 20:48:53.640614 2049 tablet_replica.cc:442] T 88412d90d7cb4239af85aff01bfab87f P 6e3aa0a8326e4168add03993b7255812: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:48:53.640816 1511 catalog_manager.cc:5495] T 88412d90d7cb4239af85aff01bfab87f P 6e3aa0a8326e4168add03993b7255812 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e3aa0a8326e4168add03993b7255812 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "6e3aa0a8326e4168add03993b7255812" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f844a7566be4416bc8dc3f9cb375536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e3aa0a8326e4168add03993b7255812" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1d2eebba7c14a698ed9ba4d04bb50d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45635 } health_report { overall_health: UNKNOWN } } }
I20260926 20:48:53.641126 2061 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 20:48:53.641247 2061 txn_status_manager.cc:929] Loading transaction status metadata into memory...
W20260926 20:48:53.649264 1995 tablet.cc:2369] T 88412d90d7cb4239af85aff01bfab87f P c1d2eebba7c14a698ed9ba4d04bb50d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:48:53.650889 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:53.647870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44837 (local address 127.0.0.1:57562)
0926 20:48:53.647953 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:53.647964 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:53.648054 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:53.648240 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:53.648241 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:53.648242 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:48:53.648288 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:53.648290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.649197 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.649198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:53.649882 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:53.649885 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.650364 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.650365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:53.650410 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:53.650424 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:48:53.650732 (+ 308us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:48:53.650742 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 20:48:53.650788 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 20:48:53.650833 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 20:48:53.650899 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:53.647908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57562 (local address 127.0.98.2:44837)
0926 20:48:53.647952 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:53.647954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:53.648001 (+ 47us) server_negotiation.cc:407] Connection header received
0926 20:48:53.648135 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:53.648137 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:53.648158 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:53.648241 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:48:53.648329 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.649091 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.649969 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.650273 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.650348 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:53.650521 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:48:53.650638 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:48:53.650713 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:53.650802 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 20:48:53.650824 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 20:48:53.652072 1947 raft_consensus.cc:1261] T 88412d90d7cb4239af85aff01bfab87f P c1d2eebba7c14a698ed9ba4d04bb50d3 [term 1 FOLLOWER]: Refusing update from remote peer 6e3aa0a8326e4168add03993b7255812: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:48:53.652245 1668 raft_consensus.cc:1261] T 88412d90d7cb4239af85aff01bfab87f P 0f844a7566be4416bc8dc3f9cb375536 [term 1 FOLLOWER]: Refusing update from remote peer 6e3aa0a8326e4168add03993b7255812: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:48:53.652281 2049 consensus_queue.cc:1035] T 88412d90d7cb4239af85aff01bfab87f P 6e3aa0a8326e4168add03993b7255812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1d2eebba7c14a698ed9ba4d04bb50d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:48:53.652529 2049 consensus_queue.cc:1035] T 88412d90d7cb4239af85aff01bfab87f P 6e3aa0a8326e4168add03993b7255812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f844a7566be4416bc8dc3f9cb375536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:48:53.652786 2024 tablet_replica.cc:439] T 88412d90d7cb4239af85aff01bfab87f P c1d2eebba7c14a698ed9ba4d04bb50d3: TxnStatusTablet state changed. Reason: New leader 6e3aa0a8326e4168add03993b7255812. Latest consensus state: current_term: 1 leader_uuid: "6e3aa0a8326e4168add03993b7255812" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f844a7566be4416bc8dc3f9cb375536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43301 } } peers { permanent_uuid: "6e3aa0a8326e4168add03993b7255812" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44837 } } peers { permanent_uuid: "c1d2eebba7c14a698ed9ba4d04bb50d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45635 } } }
I20260926 20:48:53.652864 2024 tablet_replica.cc:442] T 88412d90d7cb4239af85aff01bfab87f P c1d2eebba7c14a698ed9ba4d04bb50d3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:48:53.653189 2036 tablet_replica.cc:439] T 88412d90d7cb4239af85aff01bfab87f P 0f844a7566be4416bc8dc3f9cb375536: TxnStatusTablet state changed. Reason: New leader 6e3aa0a8326e4168add03993b7255812. Latest consensus state: current_term: 1 leader_uuid: "6e3aa0a8326e4168add03993b7255812" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f844a7566be4416bc8dc3f9cb375536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43301 } } peers { permanent_uuid: "6e3aa0a8326e4168add03993b7255812" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44837 } } peers { permanent_uuid: "c1d2eebba7c14a698ed9ba4d04bb50d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45635 } } }
I20260926 20:48:53.653287 2036 tablet_replica.cc:442] T 88412d90d7cb4239af85aff01bfab87f P 0f844a7566be4416bc8dc3f9cb375536: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:48:53.653766 2025 tablet_replica.cc:439] T 88412d90d7cb4239af85aff01bfab87f P 6e3aa0a8326e4168add03993b7255812: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6e3aa0a8326e4168add03993b7255812" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f844a7566be4416bc8dc3f9cb375536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43301 } } peers { permanent_uuid: "6e3aa0a8326e4168add03993b7255812" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44837 } } peers { permanent_uuid: "c1d2eebba7c14a698ed9ba4d04bb50d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45635 } } }
I20260926 20:48:53.653849 2025 tablet_replica.cc:442] T 88412d90d7cb4239af85aff01bfab87f P 6e3aa0a8326e4168add03993b7255812: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:48:53.653976 2036 tablet_replica.cc:439] T 88412d90d7cb4239af85aff01bfab87f P 0f844a7566be4416bc8dc3f9cb375536: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6e3aa0a8326e4168add03993b7255812" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f844a7566be4416bc8dc3f9cb375536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43301 } } peers { permanent_uuid: "6e3aa0a8326e4168add03993b7255812" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44837 } } peers { permanent_uuid: "c1d2eebba7c14a698ed9ba4d04bb50d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45635 } } }
I20260926 20:48:53.654073 2036 tablet_replica.cc:442] T 88412d90d7cb4239af85aff01bfab87f P 0f844a7566be4416bc8dc3f9cb375536: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:48:53.656260 2025 tablet_replica.cc:439] T 88412d90d7cb4239af85aff01bfab87f P 6e3aa0a8326e4168add03993b7255812: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6e3aa0a8326e4168add03993b7255812" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f844a7566be4416bc8dc3f9cb375536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43301 } } peers { permanent_uuid: "6e3aa0a8326e4168add03993b7255812" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44837 } } peers { permanent_uuid: "c1d2eebba7c14a698ed9ba4d04bb50d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45635 } } }
I20260926 20:48:53.656359 2025 tablet_replica.cc:442] T 88412d90d7cb4239af85aff01bfab87f P 6e3aa0a8326e4168add03993b7255812: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:48:53.656749 2024 tablet_replica.cc:439] T 88412d90d7cb4239af85aff01bfab87f P c1d2eebba7c14a698ed9ba4d04bb50d3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6e3aa0a8326e4168add03993b7255812" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f844a7566be4416bc8dc3f9cb375536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43301 } } peers { permanent_uuid: "6e3aa0a8326e4168add03993b7255812" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44837 } } peers { permanent_uuid: "c1d2eebba7c14a698ed9ba4d04bb50d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45635 } } }
I20260926 20:48:53.656829 2024 tablet_replica.cc:442] T 88412d90d7cb4239af85aff01bfab87f P c1d2eebba7c14a698ed9ba4d04bb50d3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:48:53.658030 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:53.655399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57564 (local address 127.0.98.2:44837)
0926 20:48:53.655433 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:53.655435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:53.655460 (+ 25us) server_negotiation.cc:407] Connection header received
0926 20:48:53.655519 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:53.655520 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:53.655529 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:53.655590 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:48:53.655665 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.656410 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.657373 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.657664 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.657732 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:53.657828 (+ 96us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:48:53.657857 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:48:53.657900 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:53.657942 (+ 42us) server_negotiation.cc:299] Negotiation successful
0926 20:48:53.657957 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 20:48:53.663936 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:53.660562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49937 (local address 127.0.98.2:44837)
0926 20:48:53.660594 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:53.660595 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:53.661311 (+ 716us) server_negotiation.cc:407] Connection header received
0926 20:48:53.661334 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:53.661335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:53.661344 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:53.661405 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:48:53.661534 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.662266 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.663101 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.663389 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.663458 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:53.663640 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:48:53.663671 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:48:53.663739 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:53.663832 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 20:48:53.663851 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 20:48:53.663949 2069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:53.660408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44837 (local address 127.0.98.2:49937)
0926 20:48:53.660546 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:53.660571 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:53.661030 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:53.661417 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:53.661418 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:53.661419 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:48:53.661470 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:53.661472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.662336 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.662337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:53.662974 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:53.662976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.663457 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.663458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:53.663510 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:53.663523 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:48:53.663730 (+ 207us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:48:53.663732 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 20:48:53.663827 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 20:48:53.663867 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":91,"threads_started":1}
I20260926 20:48:53.665887 1947 raft_consensus.cc:3037] T a577cc2345ee48fe805f5df70f82cc3c P c1d2eebba7c14a698ed9ba4d04bb50d3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:48:53.665927 1668 raft_consensus.cc:1261] T a577cc2345ee48fe805f5df70f82cc3c P 0f844a7566be4416bc8dc3f9cb375536 [term 1 FOLLOWER]: Refusing update from remote peer 6e3aa0a8326e4168add03993b7255812: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:48:53.666131 2025 consensus_queue.cc:1035] T a577cc2345ee48fe805f5df70f82cc3c P 6e3aa0a8326e4168add03993b7255812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f844a7566be4416bc8dc3f9cb375536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:48:53.666762 1947 raft_consensus.cc:1261] T a577cc2345ee48fe805f5df70f82cc3c P c1d2eebba7c14a698ed9ba4d04bb50d3 [term 1 FOLLOWER]: Refusing update from remote peer 6e3aa0a8326e4168add03993b7255812: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:48:53.666961 2025 consensus_queue.cc:1035] T a577cc2345ee48fe805f5df70f82cc3c P 6e3aa0a8326e4168add03993b7255812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1d2eebba7c14a698ed9ba4d04bb50d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:48:53.677423 1510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:48:53.677532 1510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:48:53.678177 1510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6eb902a2976241ba9a38e69de9023356: Sending DeleteTablet for 3 replicas of tablet a577cc2345ee48fe805f5df70f82cc3c
I20260926 20:48:53.678445 1648 tablet_service.cc:1514] Processing DeleteTablet for tablet a577cc2345ee48fe805f5df70f82cc3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:48:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36666
I20260926 20:48:53.678465 1928 tablet_service.cc:1514] Processing DeleteTablet for tablet a577cc2345ee48fe805f5df70f82cc3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:48:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50854
I20260926 20:48:53.678561 2042 tablet_replica.cc:330] T a577cc2345ee48fe805f5df70f82cc3c P 0f844a7566be4416bc8dc3f9cb375536: stopping tablet replica
I20260926 20:48:53.678620 2042 raft_consensus.cc:2229] T a577cc2345ee48fe805f5df70f82cc3c P 0f844a7566be4416bc8dc3f9cb375536 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:48:53.678655 2042 raft_consensus.cc:2258] T a577cc2345ee48fe805f5df70f82cc3c P 0f844a7566be4416bc8dc3f9cb375536 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:48:53.678725 2041 tablet_replica.cc:330] T a577cc2345ee48fe805f5df70f82cc3c P c1d2eebba7c14a698ed9ba4d04bb50d3: stopping tablet replica
I20260926 20:48:53.678769 2041 raft_consensus.cc:2229] T a577cc2345ee48fe805f5df70f82cc3c P c1d2eebba7c14a698ed9ba4d04bb50d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:48:53.678819 2041 raft_consensus.cc:2258] T a577cc2345ee48fe805f5df70f82cc3c P c1d2eebba7c14a698ed9ba4d04bb50d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:48:53.679169 2041 ts_tablet_manager.cc:1905] T a577cc2345ee48fe805f5df70f82cc3c P c1d2eebba7c14a698ed9ba4d04bb50d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:48:53.679292 1788 tablet_service.cc:1514] Processing DeleteTablet for tablet a577cc2345ee48fe805f5df70f82cc3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:48:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57546
I20260926 20:48:53.679823 2043 tablet_replica.cc:330] T a577cc2345ee48fe805f5df70f82cc3c P 6e3aa0a8326e4168add03993b7255812: stopping tablet replica
I20260926 20:48:53.679908 2043 raft_consensus.cc:2229] T a577cc2345ee48fe805f5df70f82cc3c P 6e3aa0a8326e4168add03993b7255812 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:48:53.679994 2043 raft_consensus.cc:2258] T a577cc2345ee48fe805f5df70f82cc3c P 6e3aa0a8326e4168add03993b7255812 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:48:53.680359 2043 ts_tablet_manager.cc:1905] T a577cc2345ee48fe805f5df70f82cc3c P 6e3aa0a8326e4168add03993b7255812: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:48:53.681228 2041 ts_tablet_manager.cc:1918] T a577cc2345ee48fe805f5df70f82cc3c P c1d2eebba7c14a698ed9ba4d04bb50d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:48:53.681286 2041 log.cc:1196] T a577cc2345ee48fe805f5df70f82cc3c P c1d2eebba7c14a698ed9ba4d04bb50d3: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-2/wal/wals/a577cc2345ee48fe805f5df70f82cc3c
I20260926 20:48:53.681510 2041 ts_tablet_manager.cc:1939] T a577cc2345ee48fe805f5df70f82cc3c P c1d2eebba7c14a698ed9ba4d04bb50d3: Deleting consensus metadata
I20260926 20:48:53.681813 1498 catalog_manager.cc:4841] TS c1d2eebba7c14a698ed9ba4d04bb50d3 (127.0.98.3:45635): tablet a577cc2345ee48fe805f5df70f82cc3c (table test-table [id=3d74af66b25442708b77cb5468246e07]) successfully deleted
I20260926 20:48:53.681921 2043 ts_tablet_manager.cc:1918] T a577cc2345ee48fe805f5df70f82cc3c P 6e3aa0a8326e4168add03993b7255812: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:48:53.681985 2043 log.cc:1196] T a577cc2345ee48fe805f5df70f82cc3c P 6e3aa0a8326e4168add03993b7255812: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-1/wal/wals/a577cc2345ee48fe805f5df70f82cc3c
I20260926 20:48:53.682212 2043 ts_tablet_manager.cc:1939] T a577cc2345ee48fe805f5df70f82cc3c P 6e3aa0a8326e4168add03993b7255812: Deleting consensus metadata
I20260926 20:48:53.682543 2042 ts_tablet_manager.cc:1905] T a577cc2345ee48fe805f5df70f82cc3c P 0f844a7566be4416bc8dc3f9cb375536: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:48:53.684114 2042 ts_tablet_manager.cc:1918] T a577cc2345ee48fe805f5df70f82cc3c P 0f844a7566be4416bc8dc3f9cb375536: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:48:53.684178 2042 log.cc:1196] T a577cc2345ee48fe805f5df70f82cc3c P 0f844a7566be4416bc8dc3f9cb375536: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455725523895-392-0/minicluster-data/ts-0/wal/wals/a577cc2345ee48fe805f5df70f82cc3c
I20260926 20:48:53.684381 2042 ts_tablet_manager.cc:1939] T a577cc2345ee48fe805f5df70f82cc3c P 0f844a7566be4416bc8dc3f9cb375536: Deleting consensus metadata
I20260926 20:48:53.684762 1497 catalog_manager.cc:4841] TS 6e3aa0a8326e4168add03993b7255812 (127.0.98.2:44837): tablet a577cc2345ee48fe805f5df70f82cc3c (table test-table [id=3d74af66b25442708b77cb5468246e07]) successfully deleted
I20260926 20:48:53.684859 1496 catalog_manager.cc:4841] TS 0f844a7566be4416bc8dc3f9cb375536 (127.0.98.1:43301): tablet a577cc2345ee48fe805f5df70f82cc3c (table test-table [id=3d74af66b25442708b77cb5468246e07]) successfully deleted
Sep 26 20:48:53 dist-test-slave-hwlj krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455733, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 20:48:53.689778 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:53.679591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36678 (local address 127.0.98.1:43301)
0926 20:48:53.679632 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:53.679634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:53.679858 (+ 224us) server_negotiation.cc:407] Connection header received
0926 20:48:53.680105 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:53.680106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:53.680119 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:53.680153 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:53.680989 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.681734 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.683003 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.683218 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.683310 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:53.686454 (+ 3144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:53.686465 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:53.686465 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:53.686473 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:53.688405 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:53.688817 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:53.688821 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:53.688822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:53.688887 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:53.689165 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:53.689166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:53.689166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:53.689349 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:53.689523 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:53.689565 (+ 42us) server_negotiation.cc:299] Negotiation successful
0926 20:48:53.689660 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 20:48:53.690065 1708 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36678
I20260926 20:48:53.698235 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:53.691320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48186 (local address 127.0.98.62:41023)
0926 20:48:53.691366 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:53.691368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:53.691404 (+ 36us) server_negotiation.cc:407] Connection header received
0926 20:48:53.691482 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:53.691483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:53.691494 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:53.691560 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:53.692184 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.692973 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.693736 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:53.694060 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:53.694126 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:53.695387 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:53.695388 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:53.695389 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:53.695397 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:53.696998 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:53.697418 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:53.697419 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:53.697419 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:53.697479 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:53.697671 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:53.697672 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:53.697672 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:53.697800 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:53.697900 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:53.698059 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 20:48:53.698136 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 20:48:53.698462 1510 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48186
I20260926 20:48:53.699131 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 1583
I20260926 20:48:53.708755 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 1723
I20260926 20:48:53.714114 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 1863
I20260926 20:48:53.718765 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 1479
2026-09-26T20:48:53Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2416 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:48:53 dist-test-slave-hwlj krb5kdc[2079](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:48:53 dist-test-slave-hwlj krb5kdc[2079](info): set up 2 sockets
Sep 26 20:48:53 dist-test-slave-hwlj krb5kdc[2079](info): commencing operation
krb5kdc: starting...
W20260926 20:48:54.755897 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/krb5kdc/test-user.keytab.
Sep 26 20:48:54 dist-test-slave-hwlj krb5kdc[2079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:48:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:48:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 20:48:54.878454 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36087
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36087
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:48:54.958446 2092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:48:54.958696 2092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:48:54.958731 2092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:48:54.959885 2092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:48:54.959959 2092 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 20:48:54.959982 2092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:48:54.960002 2092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:48:54.960033 2092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:48:54.960052 2092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:48:54.961529 2092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36087
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:48:54.961813 2092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:48:54.962034 2092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:48:54.967039 2097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:48:54.967072 2098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:48:54.967279 2092 server_base.cc:1034] running on GCE node
W20260926 20:48:54.967039 2100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:48:54.967589 2092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:48:54.967978 2092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:48:54.969187 2092 hybrid_clock.cc:648] HybridClock initialized: now 1790455734969178 us; error 69 us; skew 500 ppm
Sep 26 20:48:54 dist-test-slave-hwlj krb5kdc[2079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:48:54.972298 2092 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:48:54.972841 2092 webserver.cc:460] Webserver started at http://127.0.98.62:37155/ using document root <none> and password file <none>
I20260926 20:48:54.973114 2092 fs_manager.cc:359] Metadata directory not provided
I20260926 20:48:54.973192 2092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:48:54.973362 2092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:48:54.974529 2092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/master-0/data/instance:
uuid: "9ef56439e22a4401a988e847a088077e"
format_stamp: "Formatted at 2026-09-26 20:48:54 on dist-test-slave-hwlj"
server_key: "cf39adfca5db9c3fef5cb6fdedb4deed"
server_key_iv: "75ccf4a8077186b18ac9d7e4718f14a5"
server_key_version: "encryptionkey@0"
I20260926 20:48:54.974933 2092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/master-0/wal/instance:
uuid: "9ef56439e22a4401a988e847a088077e"
format_stamp: "Formatted at 2026-09-26 20:48:54 on dist-test-slave-hwlj"
server_key: "cf39adfca5db9c3fef5cb6fdedb4deed"
server_key_iv: "75ccf4a8077186b18ac9d7e4718f14a5"
server_key_version: "encryptionkey@0"
I20260926 20:48:54.976536 2092 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:48:54.977384 2107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:48:54.977710 2092 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:48:54.977913 2092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "9ef56439e22a4401a988e847a088077e"
format_stamp: "Formatted at 2026-09-26 20:48:54 on dist-test-slave-hwlj"
server_key: "cf39adfca5db9c3fef5cb6fdedb4deed"
server_key_iv: "75ccf4a8077186b18ac9d7e4718f14a5"
server_key_version: "encryptionkey@0"
I20260926 20:48:54.978008 2092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:48:54.990096 2092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:48:54.992650 2092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:48:54.992843 2092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:48:54.998281 2092 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36087
I20260926 20:48:54.998351 2169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36087 every 8 connection(s)
I20260926 20:48:54.998801 2092 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:48:54.999574 2170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:48:55.001884 2170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ef56439e22a4401a988e847a088077e: Bootstrap starting.
I20260926 20:48:55.002472 2170 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ef56439e22a4401a988e847a088077e: Neither blocks nor log segments found. Creating new log.
I20260926 20:48:55.002763 2170 log.cc:824] T 00000000000000000000000000000000 P 9ef56439e22a4401a988e847a088077e: Log is configured to *not* fsync() on all Append() calls
I20260926 20:48:55.003324 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 2092
I20260926 20:48:55.003396 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/master-0/wal/instance
I20260926 20:48:55.003481 2170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ef56439e22a4401a988e847a088077e: No bootstrap required, opened a new log
I20260926 20:48:55.003593 392 external_mini_cluster.cc:1428] Setting key e51387d68ff1b615c5769cd7c79ef4c7
I20260926 20:48:55.006269 2170 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ef56439e22a4401a988e847a088077e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef56439e22a4401a988e847a088077e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36087 } }
I20260926 20:48:55.006446 2170 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ef56439e22a4401a988e847a088077e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:48:55.006484 2170 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ef56439e22a4401a988e847a088077e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef56439e22a4401a988e847a088077e, State: Initialized, Role: FOLLOWER
I20260926 20:48:55.006644 2170 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ef56439e22a4401a988e847a088077e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef56439e22a4401a988e847a088077e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36087 } }
I20260926 20:48:55.006711 2170 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ef56439e22a4401a988e847a088077e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:48:55.006733 2170 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ef56439e22a4401a988e847a088077e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:48:55.006793 2170 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ef56439e22a4401a988e847a088077e [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:48:55.007413 2170 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ef56439e22a4401a988e847a088077e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef56439e22a4401a988e847a088077e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36087 } }
I20260926 20:48:55.007517 2170 leader_election.cc:302] T 00000000000000000000000000000000 P 9ef56439e22a4401a988e847a088077e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ef56439e22a4401a988e847a088077e; no voters:
I20260926 20:48:55.007686 2170 leader_election.cc:288] T 00000000000000000000000000000000 P 9ef56439e22a4401a988e847a088077e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:48:55.007802 2175 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ef56439e22a4401a988e847a088077e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:48:55.007937 2175 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ef56439e22a4401a988e847a088077e [term 1 LEADER]: Becoming Leader. State: Replica: 9ef56439e22a4401a988e847a088077e, State: Running, Role: LEADER
I20260926 20:48:55.007946 2170 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ef56439e22a4401a988e847a088077e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:48:55.009393 2175 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ef56439e22a4401a988e847a088077e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef56439e22a4401a988e847a088077e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36087 } }
I20260926 20:48:55.010082 2176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ef56439e22a4401a988e847a088077e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ef56439e22a4401a988e847a088077e. Latest consensus state: current_term: 1 leader_uuid: "9ef56439e22a4401a988e847a088077e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef56439e22a4401a988e847a088077e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36087 } } }
I20260926 20:48:55.010169 2176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ef56439e22a4401a988e847a088077e [sys.catalog]: This master's current role is: LEADER
I20260926 20:48:55.010347 2177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ef56439e22a4401a988e847a088077e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ef56439e22a4401a988e847a088077e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef56439e22a4401a988e847a088077e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36087 } } }
I20260926 20:48:55.010416 2177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ef56439e22a4401a988e847a088077e [sys.catalog]: This master's current role is: LEADER
W20260926 20:48:55.010466 2189 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9ef56439e22a4401a988e847a088077e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:48:55.010540 2189 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:48:55.010639 2191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:48:55.010854 2191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:48:55.012712 2191 catalog_manager.cc:1347] Generated new cluster ID: de92397854dd4bc88f9b4a1b47a53eb8
I20260926 20:48:55.012769 2191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 20:48:55 dist-test-slave-hwlj krb5kdc[2079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:48:55.016894 2191 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:48:55.017503 2191 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:48:55.021214 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:55.004230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44638 (local address 127.0.98.62:36087)
0926 20:48:55.004409 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:55.004420 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:55.004460 (+ 40us) server_negotiation.cc:407] Connection header received
0926 20:48:55.006233 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:55.006243 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:55.006309 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:55.006460 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:55.007265 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.008674 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.009867 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.010614 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.010727 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:55.017299 (+ 6572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:55.017303 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:55.017304 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:55.017316 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:55.019458 (+ 2142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:55.019880 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:55.019883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:55.019884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:55.019994 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:55.020205 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:55.020206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:55.020206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:55.020497 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:55.020653 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:55.020724 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 20:48:55.020987 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":69,"threads_started":1}
I20260926 20:48:55.022332 2191 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ef56439e22a4401a988e847a088077e: Generated new TSK 0
I20260926 20:48:55.022483 2191 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 20:48:55.047304 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36087
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:48:55.128952 2196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:48:55.129168 2196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:48:55.129202 2196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:48:55.130363 2196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:48:55.130410 2196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:48:55.130432 2196 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 20:48:55.130445 2196 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:48:55.130506 2196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:48:55.132093 2196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:48:55.132359 2196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:48:55.132612 2196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:48:55.138315 2202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:48:55.138296 2204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:48:55.138296 2201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:48:55.138561 2196 server_base.cc:1034] running on GCE node
I20260926 20:48:55.138928 2196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:48:55.139254 2196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:48:55.140492 2196 hybrid_clock.cc:648] HybridClock initialized: now 1790455735140468 us; error 31 us; skew 500 ppm
Sep 26 20:48:55 dist-test-slave-hwlj krb5kdc[2079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455735, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:48:55.143298 2196 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 20:48:55.143733 2196 webserver.cc:460] Webserver started at http://127.0.98.1:37347/ using document root <none> and password file <none>
I20260926 20:48:55.144045 2196 fs_manager.cc:359] Metadata directory not provided
I20260926 20:48:55.144120 2196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:48:55.144271 2196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:48:55.145350 2196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-0/data/instance:
uuid: "a2cd23ce8ed84bf397ea32956ee82f1a"
format_stamp: "Formatted at 2026-09-26 20:48:55 on dist-test-slave-hwlj"
server_key: "2020f54fcf958a7eafabe1e87e05daf8"
server_key_iv: "cb978ce47d8aed852b51971f4780ff37"
server_key_version: "encryptionkey@0"
I20260926 20:48:55.145735 2196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a2cd23ce8ed84bf397ea32956ee82f1a"
format_stamp: "Formatted at 2026-09-26 20:48:55 on dist-test-slave-hwlj"
server_key: "2020f54fcf958a7eafabe1e87e05daf8"
server_key_iv: "cb978ce47d8aed852b51971f4780ff37"
server_key_version: "encryptionkey@0"
I20260926 20:48:55.147168 2196 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:48:55.148034 2211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:48:55.148217 2196 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:48:55.148320 2196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-0/wal
uuid: "a2cd23ce8ed84bf397ea32956ee82f1a"
format_stamp: "Formatted at 2026-09-26 20:48:55 on dist-test-slave-hwlj"
server_key: "2020f54fcf958a7eafabe1e87e05daf8"
server_key_iv: "cb978ce47d8aed852b51971f4780ff37"
server_key_version: "encryptionkey@0"
I20260926 20:48:55.148403 2196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:48:55.155939 2196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:48:55.157825 2196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:48:55.157994 2196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:48:55.158489 2196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:48:55.158558 2196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:48:55.158600 2196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:48:55.158653 2196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:48:55.165467 2196 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44289
I20260926 20:48:55.165544 2326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44289 every 8 connection(s)
Sep 26 20:48:55 dist-test-slave-hwlj krb5kdc[2079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455735, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:48:55.165871 2196 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:48:55.169034 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:55.158966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58951 (local address 127.0.98.62:36087)
0926 20:48:55.159043 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:55.159046 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:55.159265 (+ 219us) server_negotiation.cc:407] Connection header received
0926 20:48:55.159736 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:55.159738 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:55.159787 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:55.159870 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:55.160640 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.161478 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.163041 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.163408 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.163487 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:55.166281 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:55.166283 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:55.166283 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:55.166291 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:55.167867 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:55.168230 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:55.168231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:55.168231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:55.168281 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:55.168531 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:55.168532 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:55.168532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:55.168637 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:55.168687 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:55.168906 (+ 219us) server_negotiation.cc:299] Negotiation successful
0926 20:48:55.168973 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 20:48:55.169207 2220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:55.158954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36087 (local address 127.0.98.1:58951)
0926 20:48:55.159132 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:55.159172 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:55.159638 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:55.159924 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:55.159927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:55.160256 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:48:55.160531 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:55.160537 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.161581 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.161582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:55.162940 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:55.162943 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.163508 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.163509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:55.163633 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:55.164225 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:48:55.164236 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:48:55.166195 (+ 1959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:48:55.167937 (+ 1742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:55.167942 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:55.167943 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:55.168159 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:55.168362 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:55.168362 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:55.168363 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:55.168472 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:55.168696 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:48:55.168700 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:48:55.168808 (+ 108us) client_negotiation.cc:769] Sending connection context
0926 20:48:55.168892 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 20:48:55.169006 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":113,"threads_started":1}
I20260926 20:48:55.169747 2327 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36087
I20260926 20:48:55.169836 2327 heartbeater.cc:464] Registering TS with master...
I20260926 20:48:55.170013 2327 heartbeater.cc:511] Master 127.0.98.62:36087 requested a full tablet report, sending...
I20260926 20:48:55.170516 2124 ts_manager.cc:194] Registered new tserver with Master: a2cd23ce8ed84bf397ea32956ee82f1a (127.0.98.1:44289)
I20260926 20:48:55.171518 2124 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58951
I20260926 20:48:55.172482 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 2196
I20260926 20:48:55.172574 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-0/wal/instance
I20260926 20:48:55.172787 392 external_mini_cluster.cc:1428] Setting key 0a0adf65e5bfa0548581cbc2542ff0d2
I20260926 20:48:55.179831 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:55.171857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60641 (local address 127.0.98.62:36087)
0926 20:48:55.171892 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:55.171893 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:55.171901 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:48:55.171919 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:55.171920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:55.171959 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:55.172039 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:55.172632 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.173540 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.174887 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.175196 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.175240 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:55.176496 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:55.176497 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:55.176497 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:55.176504 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:55.178447 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:55.178992 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:55.178993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:55.178993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:55.179050 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:55.179266 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:55.179267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:55.179267 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:55.179397 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:55.179458 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:55.179634 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 20:48:55.179716 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 20:48:55.179850 2333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:55.170806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36087 (local address 127.0.98.1:60641)
0926 20:48:55.171460 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:55.171472 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:55.171569 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:55.172048 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:55.172049 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:55.172245 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:48:55.172537 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:55.172539 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.173664 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.173665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:55.174764 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:55.174767 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.175318 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.175319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:55.175399 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:55.175982 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:48:55.175988 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:48:55.176401 (+ 413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:48:55.178544 (+ 2143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:55.178544 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:55.178545 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:55.178929 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:55.179129 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:55.179130 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:55.179130 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:55.179208 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:55.179464 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:48:55.179465 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:48:55.179508 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 20:48:55.179611 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 20:48:55.179721 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"mutex_wait_us":3,"thread_start_us":114,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 20:48:55.206629 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36087
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:48:55.293496 2336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:48:55.293704 2336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:48:55.293728 2336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:48:55.294812 2336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:48:55.294849 2336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:48:55.294870 2336 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 20:48:55.294883 2336 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:48:55.294900 2336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:48:55.296427 2336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:48:55.296622 2336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:48:55.296794 2336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:48:55.301995 2342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:48:55.302060 2341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:48:55.302003 2336 server_base.cc:1034] running on GCE node
W20260926 20:48:55.302024 2344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:48:55.302467 2336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:48:55.302727 2336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:48:55.303889 2336 hybrid_clock.cc:648] HybridClock initialized: now 1790455735303874 us; error 27 us; skew 500 ppm
Sep 26 20:48:55 dist-test-slave-hwlj krb5kdc[2079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455735, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:48:55.306385 2336 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 20:48:55.306885 2336 webserver.cc:460] Webserver started at http://127.0.98.2:34563/ using document root <none> and password file <none>
I20260926 20:48:55.307132 2336 fs_manager.cc:359] Metadata directory not provided
I20260926 20:48:55.307207 2336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:48:55.307368 2336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:48:55.308620 2336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-1/data/instance:
uuid: "8ea1b3a2a7d24919a5a031949c2cb28b"
format_stamp: "Formatted at 2026-09-26 20:48:55 on dist-test-slave-hwlj"
server_key: "9cf030f852223d8fd586538bbbe0d41a"
server_key_iv: "c334c75a6fba60f88a2300fd14d3b3e3"
server_key_version: "encryptionkey@0"
I20260926 20:48:55.309113 2336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8ea1b3a2a7d24919a5a031949c2cb28b"
format_stamp: "Formatted at 2026-09-26 20:48:55 on dist-test-slave-hwlj"
server_key: "9cf030f852223d8fd586538bbbe0d41a"
server_key_iv: "c334c75a6fba60f88a2300fd14d3b3e3"
server_key_version: "encryptionkey@0"
I20260926 20:48:55.310540 2336 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:48:55.311290 2351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:48:55.311466 2336 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:48:55.311580 2336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-1/wal
uuid: "8ea1b3a2a7d24919a5a031949c2cb28b"
format_stamp: "Formatted at 2026-09-26 20:48:55 on dist-test-slave-hwlj"
server_key: "9cf030f852223d8fd586538bbbe0d41a"
server_key_iv: "c334c75a6fba60f88a2300fd14d3b3e3"
server_key_version: "encryptionkey@0"
I20260926 20:48:55.311661 2336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:48:55.330133 2336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:48:55.332083 2336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:48:55.332250 2336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:48:55.332818 2336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:48:55.332886 2336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:48:55.332931 2336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:48:55.332964 2336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 20:48:55 dist-test-slave-hwlj krb5kdc[2079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455735, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:48:55.340281 2336 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36879
I20260926 20:48:55.340358 2466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36879 every 8 connection(s)
I20260926 20:48:55.340716 2336 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
I20260926 20:48:55.342577 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:55.333282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47103 (local address 127.0.98.62:36087)
0926 20:48:55.333356 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:55.333359 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:55.333601 (+ 242us) server_negotiation.cc:407] Connection header received
0926 20:48:55.334182 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:55.334184 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:55.334196 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:55.334274 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:55.334939 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.335658 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.336678 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.336890 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.336980 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:55.339653 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:55.339654 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:55.339654 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:55.339662 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:55.341340 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:55.341709 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:55.341710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:55.341711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:55.341768 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:55.342015 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:55.342016 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:55.342017 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:55.342136 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:55.342196 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:55.342434 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 20:48:55.342516 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 20:48:55.342710 2360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:55.333286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36087 (local address 127.0.98.2:47103)
0926 20:48:55.333467 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:55.333507 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:55.334065 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:55.334333 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:55.334336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:55.334644 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:48:55.334844 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:55.334849 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.335787 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.335788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:55.336576 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:55.336578 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.336989 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.336990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:55.337109 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:55.337790 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:48:55.337801 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:48:55.339551 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:48:55.341407 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:55.341411 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:55.341413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:55.341617 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:55.341840 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:55.341840 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:55.341840 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:55.341931 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:55.342211 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:48:55.342215 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:48:55.342320 (+ 105us) client_negotiation.cc:769] Sending connection context
0926 20:48:55.342400 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 20:48:55.342510 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":113,"threads_started":1}
I20260926 20:48:55.342940 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 2336
I20260926 20:48:55.343045 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-1/wal/instance
I20260926 20:48:55.343279 392 external_mini_cluster.cc:1428] Setting key b6da1ad2780817a5ffac79a191cafe30
I20260926 20:48:55.343298 2467 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36087
I20260926 20:48:55.343374 2467 heartbeater.cc:464] Registering TS with master...
I20260926 20:48:55.343559 2467 heartbeater.cc:511] Master 127.0.98.62:36087 requested a full tablet report, sending...
I20260926 20:48:55.343927 2124 ts_manager.cc:194] Registered new tserver with Master: 8ea1b3a2a7d24919a5a031949c2cb28b (127.0.98.2:36879)
I20260926 20:48:55.344413 2124 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47103
I20260926 20:48:55.350198 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:55.344378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38785 (local address 127.0.98.62:36087)
0926 20:48:55.344414 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:55.344416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:55.344563 (+ 147us) server_negotiation.cc:407] Connection header received
0926 20:48:55.344623 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:55.344624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:55.344633 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:55.344689 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:55.345015 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.345542 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.346286 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.346485 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.346575 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:55.347517 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:55.347518 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:55.347518 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:55.347525 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:55.348971 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:55.349352 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:55.349353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:55.349354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:55.349429 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:55.349652 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:55.349653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:55.349653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:55.349797 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:55.349873 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:55.350033 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 20:48:55.350113 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 20:48:55.350214 2473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:55.344317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36087 (local address 127.0.98.2:38785)
0926 20:48:55.344506 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:55.344518 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:55.344578 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:55.344694 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:55.344694 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:55.344866 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:48:55.344911 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:55.344912 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.345617 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.345618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:55.346176 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:55.346178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.346575 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.346576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:55.346641 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:55.347077 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:48:55.347082 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:48:55.347424 (+ 342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:48:55.349087 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:55.349088 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:55.349089 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:55.349259 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:55.349506 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:55.349506 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:55.349507 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:55.349576 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:55.349877 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:48:55.349878 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:48:55.349923 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 20:48:55.350009 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 20:48:55.350109 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":132,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 20:48:55.370510 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36087
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:48:55.454816 2476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:48:55.455070 2476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:48:55.455112 2476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:48:55.456792 2476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:48:55.456856 2476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:48:55.456892 2476 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 20:48:55.456919 2476 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:48:55.456959 2476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:48:55.458493 2476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:48:55.458772 2476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:48:55.459002 2476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:48:55.464167 2481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:48:55.464139 2482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:48:55.464146 2484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:48:55.464340 2476 server_base.cc:1034] running on GCE node
I20260926 20:48:55.464614 2476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:48:55.464861 2476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:48:55.466053 2476 hybrid_clock.cc:648] HybridClock initialized: now 1790455735466038 us; error 26 us; skew 500 ppm
Sep 26 20:48:55 dist-test-slave-hwlj krb5kdc[2079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455735, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:48:55.468683 2476 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 20:48:55.469157 2476 webserver.cc:460] Webserver started at http://127.0.98.3:41295/ using document root <none> and password file <none>
I20260926 20:48:55.469408 2476 fs_manager.cc:359] Metadata directory not provided
I20260926 20:48:55.469491 2476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:48:55.469669 2476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:48:55.470990 2476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-2/data/instance:
uuid: "0658d0cafe6c4e768d0b016696a5277e"
format_stamp: "Formatted at 2026-09-26 20:48:55 on dist-test-slave-hwlj"
server_key: "17487df64597ef6f4d5efb63b69eecd8"
server_key_iv: "2ca5b63df39490c637524ddaaf8436f1"
server_key_version: "encryptionkey@0"
I20260926 20:48:55.471421 2476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0658d0cafe6c4e768d0b016696a5277e"
format_stamp: "Formatted at 2026-09-26 20:48:55 on dist-test-slave-hwlj"
server_key: "17487df64597ef6f4d5efb63b69eecd8"
server_key_iv: "2ca5b63df39490c637524ddaaf8436f1"
server_key_version: "encryptionkey@0"
I20260926 20:48:55.473045 2476 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260926 20:48:55.473788 2491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:48:55.474020 2476 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:48:55.474092 2476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-2/wal
uuid: "0658d0cafe6c4e768d0b016696a5277e"
format_stamp: "Formatted at 2026-09-26 20:48:55 on dist-test-slave-hwlj"
server_key: "17487df64597ef6f4d5efb63b69eecd8"
server_key_iv: "2ca5b63df39490c637524ddaaf8436f1"
server_key_version: "encryptionkey@0"
I20260926 20:48:55.474229 2476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:48:55.491560 2476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:48:55.493762 2476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:48:55.493965 2476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:48:55.494505 2476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:48:55.494542 2476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:48:55.494565 2476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:48:55.494577 2476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:48:55.501349 2476 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42145
I20260926 20:48:55.501467 2606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42145 every 8 connection(s)
I20260926 20:48:55.501758 2476 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 20:48:55 dist-test-slave-hwlj krb5kdc[2079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455735, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:48:55.505330 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:55.495145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35459 (local address 127.0.98.62:36087)
0926 20:48:55.495194 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:55.495197 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:55.495512 (+ 315us) server_negotiation.cc:407] Connection header received
0926 20:48:55.496136 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:55.496138 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:55.496152 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:55.496226 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:55.497092 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.497929 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.499506 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.499860 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.499944 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:55.502413 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:55.502414 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:55.502414 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:55.502420 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:55.504130 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:55.504511 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:55.504512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:55.504512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:55.504572 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:55.504786 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:55.504787 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:55.504787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:55.504899 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:55.504960 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:55.505200 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 20:48:55.505269 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 20:48:55.505400 2500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:55.495163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36087 (local address 127.0.98.3:35459)
0926 20:48:55.495347 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:55.495382 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:55.496020 (+ 638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:55.496310 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:55.496315 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:55.496677 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:48:55.496978 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:55.496984 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.498042 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.498043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:55.499400 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:55.499403 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.499956 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.499957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:55.500098 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:55.500676 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:48:55.500704 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:48:55.502307 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:48:55.504200 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:55.504207 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:55.504209 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:55.504434 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:55.504626 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:55.504627 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:55.504627 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:55.504721 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:55.504952 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:48:55.504954 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:48:55.505059 (+ 105us) client_negotiation.cc:769] Sending connection context
0926 20:48:55.505135 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 20:48:55.505250 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":100,"threads_started":1}
I20260926 20:48:55.505728 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 2476
I20260926 20:48:55.505831 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-2/wal/instance
I20260926 20:48:55.506042 392 external_mini_cluster.cc:1428] Setting key 3d6257dc6fbdc5456774d1499cb4c6f2
I20260926 20:48:55.507050 2607 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36087
I20260926 20:48:55.507138 2607 heartbeater.cc:464] Registering TS with master...
I20260926 20:48:55.507301 2607 heartbeater.cc:511] Master 127.0.98.62:36087 requested a full tablet report, sending...
I20260926 20:48:55.507628 2123 ts_manager.cc:194] Registered new tserver with Master: 0658d0cafe6c4e768d0b016696a5277e (127.0.98.3:42145)
I20260926 20:48:55.508041 2123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35459
I20260926 20:48:55.512370 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:55.506996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45595 (local address 127.0.98.62:36087)
0926 20:48:55.507036 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:55.507038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:55.507232 (+ 194us) server_negotiation.cc:407] Connection header received
0926 20:48:55.507288 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:55.507288 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:55.507298 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:55.507356 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:55.507728 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.508354 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.509185 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.509404 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.509462 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:55.510316 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:55.510317 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:55.510318 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:55.510323 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:55.511538 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:55.511885 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:55.511886 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:55.511887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:55.511933 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:55.512050 (+ 117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:55.512050 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:55.512050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:55.512130 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:55.512173 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:55.512267 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 20:48:55.512316 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 20:48:55.512387 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:55.506937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36087 (local address 127.0.98.3:45595)
0926 20:48:55.507176 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:55.507188 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:55.507243 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:55.507377 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:55.507378 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:55.507570 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:48:55.507652 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:55.507653 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.508449 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.508451 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:55.509110 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:55.509112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.509464 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.509465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:55.509511 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:55.509952 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:48:55.509956 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:48:55.510249 (+ 293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:48:55.511610 (+ 1361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:55.511610 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:55.511611 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:55.511829 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:55.511973 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:55.511973 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:55.511974 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:55.512017 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:55.512174 (+ 157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:48:55.512175 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:48:55.512207 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 20:48:55.512253 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 20:48:55.512322 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":138,"threads_started":1}
I20260926 20:48:55.517696 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 20:48:55 dist-test-slave-hwlj krb5kdc[2079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 20:48:55 dist-test-slave-hwlj krb5kdc[2079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:48:55.539873 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:55.532761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44652 (local address 127.0.98.62:36087)
0926 20:48:55.532838 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:55.532840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:55.532849 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:48:55.532891 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:55.532891 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:55.532900 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:55.532951 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:55.533502 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.534000 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.534804 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.534977 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.535026 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:55.537339 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:55.537340 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:55.537340 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:55.537347 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:55.538697 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:55.539086 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:55.539087 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:55.539087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:55.539137 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:55.539319 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:55.539320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:55.539320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:55.539418 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:55.539492 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:55.539672 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 20:48:55.539737 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 20:48:55.541658 2123 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44652:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:48:55.543164 2123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:48:55.549113 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:55.545553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44289 (local address 127.0.0.1:50812)
0926 20:48:55.545704 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:55.545744 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:55.545809 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:55.546083 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:55.546086 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:55.546088 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:48:55.546158 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:55.546160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.547117 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.547118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:55.548084 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:55.548085 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.548816 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.548817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:55.548908 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:55.548924 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 20:48:55.548992 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 20:48:55.549040 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":45,"threads_started":1}
I20260926 20:48:55.549157 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:55.545675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50812 (local address 127.0.98.1:44289)
0926 20:48:55.545850 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:55.545855 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:55.545866 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:48:55.545907 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:55.545909 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:55.545925 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:55.546073 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:48:55.546241 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.547017 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.548174 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.548733 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.548793 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:55.548863 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:55.549020 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 20:48:55.549061 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":72,"threads_started":1}
I20260926 20:48:55.549780 2255 tablet_service.cc:1467] Processing CreateTablet for tablet dd515d26be474ab3960a9a3f039f20e4 (DEFAULT_TABLE table=test-table [id=c87fb36613de426db1ff3bcd94d05218]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:48:55.550153 2255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd515d26be474ab3960a9a3f039f20e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:48:55.550308 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:55.545739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36879 (local address 127.0.0.1:58702)
0926 20:48:55.546295 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:55.546305 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:55.546353 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:55.546580 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:55.546581 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:55.546583 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:48:55.546620 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:55.546621 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.547863 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.547864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:55.548759 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:55.548761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.550079 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.550080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:55.550139 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:55.550143 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:48:55.550205 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 20:48:55.550242 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":82,"threads_started":1}
I20260926 20:48:55.550688 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:55.545836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58702 (local address 127.0.98.2:36879)
0926 20:48:55.546006 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:55.546011 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:55.546350 (+ 339us) server_negotiation.cc:407] Connection header received
0926 20:48:55.546420 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:55.546422 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:55.546438 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:55.546504 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:48:55.546695 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.547783 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.548880 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.549921 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.550080 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:55.550141 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:55.550303 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 20:48:55.550342 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":77,"threads_started":1}
I20260926 20:48:55.550911 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:55.545563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42145 (local address 127.0.0.1:57502)
0926 20:48:55.546020 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:55.546031 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:55.546073 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:55.546505 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:55.546506 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:55.546508 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:48:55.546556 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:55.546557 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.548875 (+ 2318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.548876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:55.549708 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:55.549710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.550668 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.550669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:55.550758 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:55.550763 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:48:55.550805 (+ 42us) client_negotiation.cc:240] Negotiation successful
0926 20:48:55.550838 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":91,"threads_started":1}
I20260926 20:48:55.551249 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:55.545886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57502 (local address 127.0.98.3:42145)
0926 20:48:55.546093 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:55.546096 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:55.546108 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:48:55.546142 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:55.546144 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:55.546159 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:55.546451 (+ 292us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:48:55.546660 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.547787 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.549802 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.550535 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.551061 (+ 526us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:55.551121 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:55.551151 (+ 30us) server_negotiation.cc:299] Negotiation successful
0926 20:48:55.551180 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":76,"threads_started":1}
I20260926 20:48:55.551455 2401 tablet_service.cc:1467] Processing CreateTablet for tablet dd515d26be474ab3960a9a3f039f20e4 (DEFAULT_TABLE table=test-table [id=c87fb36613de426db1ff3bcd94d05218]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:48:55.551836 2401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd515d26be474ab3960a9a3f039f20e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:48:55.552307 2541 tablet_service.cc:1467] Processing CreateTablet for tablet dd515d26be474ab3960a9a3f039f20e4 (DEFAULT_TABLE table=test-table [id=c87fb36613de426db1ff3bcd94d05218]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:48:55.552608 2541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd515d26be474ab3960a9a3f039f20e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:48:55.554028 2630 tablet_bootstrap.cc:492] T dd515d26be474ab3960a9a3f039f20e4 P a2cd23ce8ed84bf397ea32956ee82f1a: Bootstrap starting.
I20260926 20:48:55.554708 2630 tablet_bootstrap.cc:654] T dd515d26be474ab3960a9a3f039f20e4 P a2cd23ce8ed84bf397ea32956ee82f1a: Neither blocks nor log segments found. Creating new log.
I20260926 20:48:55.555042 2630 log.cc:824] T dd515d26be474ab3960a9a3f039f20e4 P a2cd23ce8ed84bf397ea32956ee82f1a: Log is configured to *not* fsync() on all Append() calls
I20260926 20:48:55.555523 2631 tablet_bootstrap.cc:492] T dd515d26be474ab3960a9a3f039f20e4 P 8ea1b3a2a7d24919a5a031949c2cb28b: Bootstrap starting.
I20260926 20:48:55.555549 2633 tablet_bootstrap.cc:492] T dd515d26be474ab3960a9a3f039f20e4 P 0658d0cafe6c4e768d0b016696a5277e: Bootstrap starting.
I20260926 20:48:55.555843 2630 tablet_bootstrap.cc:492] T dd515d26be474ab3960a9a3f039f20e4 P a2cd23ce8ed84bf397ea32956ee82f1a: No bootstrap required, opened a new log
I20260926 20:48:55.555908 2630 ts_tablet_manager.cc:1397] T dd515d26be474ab3960a9a3f039f20e4 P a2cd23ce8ed84bf397ea32956ee82f1a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:48:55.556173 2631 tablet_bootstrap.cc:654] T dd515d26be474ab3960a9a3f039f20e4 P 8ea1b3a2a7d24919a5a031949c2cb28b: Neither blocks nor log segments found. Creating new log.
I20260926 20:48:55.556324 2633 tablet_bootstrap.cc:654] T dd515d26be474ab3960a9a3f039f20e4 P 0658d0cafe6c4e768d0b016696a5277e: Neither blocks nor log segments found. Creating new log.
I20260926 20:48:55.556468 2631 log.cc:824] T dd515d26be474ab3960a9a3f039f20e4 P 8ea1b3a2a7d24919a5a031949c2cb28b: Log is configured to *not* fsync() on all Append() calls
I20260926 20:48:55.556689 2633 log.cc:824] T dd515d26be474ab3960a9a3f039f20e4 P 0658d0cafe6c4e768d0b016696a5277e: Log is configured to *not* fsync() on all Append() calls
I20260926 20:48:55.557206 2631 tablet_bootstrap.cc:492] T dd515d26be474ab3960a9a3f039f20e4 P 8ea1b3a2a7d24919a5a031949c2cb28b: No bootstrap required, opened a new log
I20260926 20:48:55.557265 2631 ts_tablet_manager.cc:1397] T dd515d26be474ab3960a9a3f039f20e4 P 8ea1b3a2a7d24919a5a031949c2cb28b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:48:55.557953 2630 raft_consensus.cc:348] T dd515d26be474ab3960a9a3f039f20e4 P a2cd23ce8ed84bf397ea32956ee82f1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } } peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } }
I20260926 20:48:55.558109 2630 raft_consensus.cc:374] T dd515d26be474ab3960a9a3f039f20e4 P a2cd23ce8ed84bf397ea32956ee82f1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:48:55.558144 2630 raft_consensus.cc:729] T dd515d26be474ab3960a9a3f039f20e4 P a2cd23ce8ed84bf397ea32956ee82f1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2cd23ce8ed84bf397ea32956ee82f1a, State: Initialized, Role: FOLLOWER
I20260926 20:48:55.558259 2630 consensus_queue.cc:260] T dd515d26be474ab3960a9a3f039f20e4 P a2cd23ce8ed84bf397ea32956ee82f1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } } peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } }
I20260926 20:48:55.558517 2630 ts_tablet_manager.cc:1428] T dd515d26be474ab3960a9a3f039f20e4 P a2cd23ce8ed84bf397ea32956ee82f1a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:48:55.558570 2327 heartbeater.cc:503] Master 127.0.98.62:36087 was elected leader, sending a full tablet report...
I20260926 20:48:55.559229 2631 raft_consensus.cc:348] T dd515d26be474ab3960a9a3f039f20e4 P 8ea1b3a2a7d24919a5a031949c2cb28b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } } peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } }
I20260926 20:48:55.559413 2631 raft_consensus.cc:374] T dd515d26be474ab3960a9a3f039f20e4 P 8ea1b3a2a7d24919a5a031949c2cb28b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:48:55.559463 2631 raft_consensus.cc:729] T dd515d26be474ab3960a9a3f039f20e4 P 8ea1b3a2a7d24919a5a031949c2cb28b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ea1b3a2a7d24919a5a031949c2cb28b, State: Initialized, Role: FOLLOWER
I20260926 20:48:55.559579 2631 consensus_queue.cc:260] T dd515d26be474ab3960a9a3f039f20e4 P 8ea1b3a2a7d24919a5a031949c2cb28b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } } peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } }
I20260926 20:48:55.559863 2631 ts_tablet_manager.cc:1428] T dd515d26be474ab3960a9a3f039f20e4 P 8ea1b3a2a7d24919a5a031949c2cb28b: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260926 20:48:55.559890 2467 heartbeater.cc:503] Master 127.0.98.62:36087 was elected leader, sending a full tablet report...
I20260926 20:48:55.560310 2633 tablet_bootstrap.cc:492] T dd515d26be474ab3960a9a3f039f20e4 P 0658d0cafe6c4e768d0b016696a5277e: No bootstrap required, opened a new log
I20260926 20:48:55.560393 2633 ts_tablet_manager.cc:1397] T dd515d26be474ab3960a9a3f039f20e4 P 0658d0cafe6c4e768d0b016696a5277e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260926 20:48:55.561861 2633 raft_consensus.cc:348] T dd515d26be474ab3960a9a3f039f20e4 P 0658d0cafe6c4e768d0b016696a5277e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } } peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } }
I20260926 20:48:55.561977 2633 raft_consensus.cc:374] T dd515d26be474ab3960a9a3f039f20e4 P 0658d0cafe6c4e768d0b016696a5277e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:48:55.561998 2633 raft_consensus.cc:729] T dd515d26be474ab3960a9a3f039f20e4 P 0658d0cafe6c4e768d0b016696a5277e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0658d0cafe6c4e768d0b016696a5277e, State: Initialized, Role: FOLLOWER
I20260926 20:48:55.562104 2633 consensus_queue.cc:260] T dd515d26be474ab3960a9a3f039f20e4 P 0658d0cafe6c4e768d0b016696a5277e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } } peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } }
I20260926 20:48:55.562310 2633 ts_tablet_manager.cc:1428] T dd515d26be474ab3960a9a3f039f20e4 P 0658d0cafe6c4e768d0b016696a5277e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:48:55.562374 2607 heartbeater.cc:503] Master 127.0.98.62:36087 was elected leader, sending a full tablet report...
W20260926 20:48:55.591310 2468 tablet.cc:2369] T dd515d26be474ab3960a9a3f039f20e4 P 8ea1b3a2a7d24919a5a031949c2cb28b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:48:55.666683 2328 tablet.cc:2369] T dd515d26be474ab3960a9a3f039f20e4 P a2cd23ce8ed84bf397ea32956ee82f1a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:48:55.752331 2608 tablet.cc:2369] T dd515d26be474ab3960a9a3f039f20e4 P 0658d0cafe6c4e768d0b016696a5277e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:48:55.782763 2638 raft_consensus.cc:482] T dd515d26be474ab3960a9a3f039f20e4 P 0658d0cafe6c4e768d0b016696a5277e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:48:55.782938 2638 raft_consensus.cc:504] T dd515d26be474ab3960a9a3f039f20e4 P 0658d0cafe6c4e768d0b016696a5277e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } } peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } }
I20260926 20:48:55.783308 2638 leader_election.cc:288] T dd515d26be474ab3960a9a3f039f20e4 P 0658d0cafe6c4e768d0b016696a5277e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2cd23ce8ed84bf397ea32956ee82f1a (127.0.98.1:44289), 8ea1b3a2a7d24919a5a031949c2cb28b (127.0.98.2:36879)
I20260926 20:48:55.786650 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:55.783717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38037 (local address 127.0.98.1:44289)
0926 20:48:55.783796 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:55.783799 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:55.783813 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:48:55.783896 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:55.783898 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:55.783912 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:55.783991 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:48:55.784245 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.784776 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.785775 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.786180 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.786236 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:55.786244 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:55.786524 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 20:48:55.786579 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 20:48:55.786675 2500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:55.783658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44289 (local address 127.0.98.3:38037)
0926 20:48:55.783704 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:55.783717 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:55.783799 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:55.783996 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:55.783997 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:55.784000 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:48:55.784145 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:55.784148 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.784878 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.784879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:55.785678 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:55.785680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.786271 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.786272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:55.786347 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:55.786350 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:48:55.786400 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 20:48:55.786597 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260926 20:48:55.786711 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:55.783929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58677 (local address 127.0.98.2:36879)
0926 20:48:55.783980 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:55.783982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:55.784121 (+ 139us) server_negotiation.cc:407] Connection header received
0926 20:48:55.784151 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:55.784152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:55.784166 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:55.784227 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:48:55.784400 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.784901 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.785849 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.786206 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.786304 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:55.786313 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:55.786602 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 20:48:55.786638 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 20:48:55.786813 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:55.783808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36879 (local address 127.0.98.3:58677)
0926 20:48:55.783990 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:55.784002 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:55.784101 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:55.784245 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:55.784245 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:55.784247 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:48:55.784312 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:55.784313 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.784992 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.784993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:55.785757 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:55.785760 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.786322 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.786323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:55.786378 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:55.786514 (+ 136us) client_negotiation.cc:769] Sending connection context
0926 20:48:55.786576 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 20:48:55.786627 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":13,"thread_start_us":106,"threads_started":1}
I20260926 20:48:55.787134 2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd515d26be474ab3960a9a3f039f20e4" candidate_uuid: "0658d0cafe6c4e768d0b016696a5277e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" is_pre_election: true
I20260926 20:48:55.787142 2421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd515d26be474ab3960a9a3f039f20e4" candidate_uuid: "0658d0cafe6c4e768d0b016696a5277e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" is_pre_election: true
I20260926 20:48:55.787317 2281 raft_consensus.cc:2454] T dd515d26be474ab3960a9a3f039f20e4 P a2cd23ce8ed84bf397ea32956ee82f1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0658d0cafe6c4e768d0b016696a5277e in term 0.
I20260926 20:48:55.787333 2421 raft_consensus.cc:2454] T dd515d26be474ab3960a9a3f039f20e4 P 8ea1b3a2a7d24919a5a031949c2cb28b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0658d0cafe6c4e768d0b016696a5277e in term 0.
I20260926 20:48:55.787565 2495 leader_election.cc:302] T dd515d26be474ab3960a9a3f039f20e4 P 0658d0cafe6c4e768d0b016696a5277e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0658d0cafe6c4e768d0b016696a5277e, a2cd23ce8ed84bf397ea32956ee82f1a; no voters:
I20260926 20:48:55.787783 2638 raft_consensus.cc:2781] T dd515d26be474ab3960a9a3f039f20e4 P 0658d0cafe6c4e768d0b016696a5277e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:48:55.787866 2638 raft_consensus.cc:482] T dd515d26be474ab3960a9a3f039f20e4 P 0658d0cafe6c4e768d0b016696a5277e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:48:55.787899 2638 raft_consensus.cc:3037] T dd515d26be474ab3960a9a3f039f20e4 P 0658d0cafe6c4e768d0b016696a5277e [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:48:55.788965 2638 raft_consensus.cc:504] T dd515d26be474ab3960a9a3f039f20e4 P 0658d0cafe6c4e768d0b016696a5277e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } } peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } }
I20260926 20:48:55.789142 2638 leader_election.cc:288] T dd515d26be474ab3960a9a3f039f20e4 P 0658d0cafe6c4e768d0b016696a5277e [CANDIDATE]: Term 1 election: Requested vote from peers a2cd23ce8ed84bf397ea32956ee82f1a (127.0.98.1:44289), 8ea1b3a2a7d24919a5a031949c2cb28b (127.0.98.2:36879)
I20260926 20:48:55.789353 2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd515d26be474ab3960a9a3f039f20e4" candidate_uuid: "0658d0cafe6c4e768d0b016696a5277e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a"
I20260926 20:48:55.789402 2421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd515d26be474ab3960a9a3f039f20e4" candidate_uuid: "0658d0cafe6c4e768d0b016696a5277e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b"
I20260926 20:48:55.789474 2281 raft_consensus.cc:3037] T dd515d26be474ab3960a9a3f039f20e4 P a2cd23ce8ed84bf397ea32956ee82f1a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:48:55.789510 2421 raft_consensus.cc:3037] T dd515d26be474ab3960a9a3f039f20e4 P 8ea1b3a2a7d24919a5a031949c2cb28b [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:48:55.790184 2281 raft_consensus.cc:2454] T dd515d26be474ab3960a9a3f039f20e4 P a2cd23ce8ed84bf397ea32956ee82f1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0658d0cafe6c4e768d0b016696a5277e in term 1.
I20260926 20:48:55.790211 2421 raft_consensus.cc:2454] T dd515d26be474ab3960a9a3f039f20e4 P 8ea1b3a2a7d24919a5a031949c2cb28b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0658d0cafe6c4e768d0b016696a5277e in term 1.
I20260926 20:48:55.790403 2495 leader_election.cc:302] T dd515d26be474ab3960a9a3f039f20e4 P 0658d0cafe6c4e768d0b016696a5277e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0658d0cafe6c4e768d0b016696a5277e, 8ea1b3a2a7d24919a5a031949c2cb28b; no voters:
I20260926 20:48:55.790529 2638 raft_consensus.cc:2781] T dd515d26be474ab3960a9a3f039f20e4 P 0658d0cafe6c4e768d0b016696a5277e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:48:55.790748 2638 raft_consensus.cc:686] T dd515d26be474ab3960a9a3f039f20e4 P 0658d0cafe6c4e768d0b016696a5277e [term 1 LEADER]: Becoming Leader. State: Replica: 0658d0cafe6c4e768d0b016696a5277e, State: Running, Role: LEADER
I20260926 20:48:55.790861 2638 consensus_queue.cc:237] T dd515d26be474ab3960a9a3f039f20e4 P 0658d0cafe6c4e768d0b016696a5277e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } } peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } }
I20260926 20:48:55.791759 2123 catalog_manager.cc:5495] T dd515d26be474ab3960a9a3f039f20e4 P 0658d0cafe6c4e768d0b016696a5277e reported cstate change: term changed from 0 to 1, leader changed from <none> to 0658d0cafe6c4e768d0b016696a5277e (127.0.98.3). New cstate: current_term: 1 leader_uuid: "0658d0cafe6c4e768d0b016696a5277e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } health_report { overall_health: UNKNOWN } } }
Sep 26 20:48:55 dist-test-slave-hwlj krb5kdc[2079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:48:55.855638 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:55.848599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44662 (local address 127.0.98.62:36087)
0926 20:48:55.848687 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:55.848689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:55.848817 (+ 128us) server_negotiation.cc:407] Connection header received
0926 20:48:55.848902 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:55.848903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:55.848917 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:55.848980 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:55.849444 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.849928 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.850749 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.850927 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.851005 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:55.853163 (+ 2158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:55.853165 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:55.853165 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:55.853172 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:55.854396 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:55.854828 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:55.854829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:55.854829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:55.854883 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:55.855086 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:55.855086 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:55.855087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:55.855197 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:55.855274 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:55.855429 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 20:48:55.855510 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 20:48:55.855731 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:55.848474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36087 (local address 127.0.0.1:44662)
0926 20:48:55.848731 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:55.848748 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:55.848839 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:55.848983 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:55.848984 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:55.849309 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:48:55.849353 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:55.849356 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.850012 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.850014 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:55.850645 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:55.850647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.851002 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.851003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:55.851057 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:55.851637 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:48:55.851642 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:48:55.853070 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:48:55.854521 (+ 1451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:55.854528 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:55.854531 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:55.854734 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:55.854951 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:48:55.854951 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:48:55.854952 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:48:55.855007 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:48:55.855266 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:48:55.855272 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:48:55.855318 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 20:48:55.855406 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 20:48:55.855588 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":82,"thread_start_us":133,"threads_started":1}
I20260926 20:48:55.857635 2123 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44662:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 20:48:55.857813 2123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:48:55.859346 2255 tablet_service.cc:1467] Processing CreateTablet for tablet 404e92da201544bd832722c66b5f39bc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=65ff4a1ca82d4e4da64c1180a3e18af4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:48:55.859336 2401 tablet_service.cc:1467] Processing CreateTablet for tablet 404e92da201544bd832722c66b5f39bc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=65ff4a1ca82d4e4da64c1180a3e18af4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:48:55.859386 2541 tablet_service.cc:1467] Processing CreateTablet for tablet 404e92da201544bd832722c66b5f39bc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=65ff4a1ca82d4e4da64c1180a3e18af4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:48:55.859508 2401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 404e92da201544bd832722c66b5f39bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:48:55.859508 2255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 404e92da201544bd832722c66b5f39bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:48:55.859560 2541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 404e92da201544bd832722c66b5f39bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:48:55.861033 2630 tablet_bootstrap.cc:492] T 404e92da201544bd832722c66b5f39bc P a2cd23ce8ed84bf397ea32956ee82f1a: Bootstrap starting.
I20260926 20:48:55.861049 2631 tablet_bootstrap.cc:492] T 404e92da201544bd832722c66b5f39bc P 8ea1b3a2a7d24919a5a031949c2cb28b: Bootstrap starting.
I20260926 20:48:55.861078 2633 tablet_bootstrap.cc:492] T 404e92da201544bd832722c66b5f39bc P 0658d0cafe6c4e768d0b016696a5277e: Bootstrap starting.
I20260926 20:48:55.861418 2630 tablet_bootstrap.cc:654] T 404e92da201544bd832722c66b5f39bc P a2cd23ce8ed84bf397ea32956ee82f1a: Neither blocks nor log segments found. Creating new log.
I20260926 20:48:55.861441 2631 tablet_bootstrap.cc:654] T 404e92da201544bd832722c66b5f39bc P 8ea1b3a2a7d24919a5a031949c2cb28b: Neither blocks nor log segments found. Creating new log.
I20260926 20:48:55.861455 2633 tablet_bootstrap.cc:654] T 404e92da201544bd832722c66b5f39bc P 0658d0cafe6c4e768d0b016696a5277e: Neither blocks nor log segments found. Creating new log.
I20260926 20:48:55.862005 2630 tablet_bootstrap.cc:492] T 404e92da201544bd832722c66b5f39bc P a2cd23ce8ed84bf397ea32956ee82f1a: No bootstrap required, opened a new log
I20260926 20:48:55.862059 2630 ts_tablet_manager.cc:1397] T 404e92da201544bd832722c66b5f39bc P a2cd23ce8ed84bf397ea32956ee82f1a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:48:55.862263 2633 tablet_bootstrap.cc:492] T 404e92da201544bd832722c66b5f39bc P 0658d0cafe6c4e768d0b016696a5277e: No bootstrap required, opened a new log
I20260926 20:48:55.862253 2631 tablet_bootstrap.cc:492] T 404e92da201544bd832722c66b5f39bc P 8ea1b3a2a7d24919a5a031949c2cb28b: No bootstrap required, opened a new log
I20260926 20:48:55.862244 2630 raft_consensus.cc:348] T 404e92da201544bd832722c66b5f39bc P a2cd23ce8ed84bf397ea32956ee82f1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } } peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } }
I20260926 20:48:55.862313 2631 ts_tablet_manager.cc:1397] T 404e92da201544bd832722c66b5f39bc P 8ea1b3a2a7d24919a5a031949c2cb28b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260926 20:48:55.862313 2633 ts_tablet_manager.cc:1397] T 404e92da201544bd832722c66b5f39bc P 0658d0cafe6c4e768d0b016696a5277e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:48:55.862320 2630 raft_consensus.cc:374] T 404e92da201544bd832722c66b5f39bc P a2cd23ce8ed84bf397ea32956ee82f1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:48:55.862346 2630 raft_consensus.cc:729] T 404e92da201544bd832722c66b5f39bc P a2cd23ce8ed84bf397ea32956ee82f1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2cd23ce8ed84bf397ea32956ee82f1a, State: Initialized, Role: FOLLOWER
I20260926 20:48:55.862392 2630 consensus_queue.cc:260] T 404e92da201544bd832722c66b5f39bc P a2cd23ce8ed84bf397ea32956ee82f1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } } peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } }
I20260926 20:48:55.862462 2631 raft_consensus.cc:348] T 404e92da201544bd832722c66b5f39bc P 8ea1b3a2a7d24919a5a031949c2cb28b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } } peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } }
I20260926 20:48:55.862450 2633 raft_consensus.cc:348] T 404e92da201544bd832722c66b5f39bc P 0658d0cafe6c4e768d0b016696a5277e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } } peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } }
I20260926 20:48:55.862506 2630 ts_tablet_manager.cc:1428] T 404e92da201544bd832722c66b5f39bc P a2cd23ce8ed84bf397ea32956ee82f1a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 20:48:55.862533 2631 raft_consensus.cc:374] T 404e92da201544bd832722c66b5f39bc P 8ea1b3a2a7d24919a5a031949c2cb28b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:48:55.862533 2633 raft_consensus.cc:374] T 404e92da201544bd832722c66b5f39bc P 0658d0cafe6c4e768d0b016696a5277e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:48:55.862564 2631 raft_consensus.cc:729] T 404e92da201544bd832722c66b5f39bc P 8ea1b3a2a7d24919a5a031949c2cb28b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ea1b3a2a7d24919a5a031949c2cb28b, State: Initialized, Role: FOLLOWER
I20260926 20:48:55.862571 2633 raft_consensus.cc:729] T 404e92da201544bd832722c66b5f39bc P 0658d0cafe6c4e768d0b016696a5277e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0658d0cafe6c4e768d0b016696a5277e, State: Initialized, Role: FOLLOWER
I20260926 20:48:55.862607 2631 consensus_queue.cc:260] T 404e92da201544bd832722c66b5f39bc P 8ea1b3a2a7d24919a5a031949c2cb28b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } } peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } }
I20260926 20:48:55.862612 2633 consensus_queue.cc:260] T 404e92da201544bd832722c66b5f39bc P 0658d0cafe6c4e768d0b016696a5277e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } } peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } }
I20260926 20:48:55.862625 2636 tablet_replica.cc:439] T 404e92da201544bd832722c66b5f39bc P a2cd23ce8ed84bf397ea32956ee82f1a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } } peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } } }
I20260926 20:48:55.862701 2633 ts_tablet_manager.cc:1428] T 404e92da201544bd832722c66b5f39bc P 0658d0cafe6c4e768d0b016696a5277e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 20:48:55.862699 2636 tablet_replica.cc:442] T 404e92da201544bd832722c66b5f39bc P a2cd23ce8ed84bf397ea32956ee82f1a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:48:55.862717 2631 ts_tablet_manager.cc:1428] T 404e92da201544bd832722c66b5f39bc P 8ea1b3a2a7d24919a5a031949c2cb28b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:48:55.862768 2637 tablet_replica.cc:439] T 404e92da201544bd832722c66b5f39bc P 8ea1b3a2a7d24919a5a031949c2cb28b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } } peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } } }
I20260926 20:48:55.862850 2637 tablet_replica.cc:442] T 404e92da201544bd832722c66b5f39bc P 8ea1b3a2a7d24919a5a031949c2cb28b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:48:55.862825 2638 tablet_replica.cc:439] T 404e92da201544bd832722c66b5f39bc P 0658d0cafe6c4e768d0b016696a5277e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } } peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } } }
I20260926 20:48:55.862902 2638 tablet_replica.cc:442] T 404e92da201544bd832722c66b5f39bc P 0658d0cafe6c4e768d0b016696a5277e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:48:55.898176 2636 raft_consensus.cc:482] T 404e92da201544bd832722c66b5f39bc P a2cd23ce8ed84bf397ea32956ee82f1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:48:55.898277 2636 raft_consensus.cc:504] T 404e92da201544bd832722c66b5f39bc P a2cd23ce8ed84bf397ea32956ee82f1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } } peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } }
I20260926 20:48:55.898681 2636 leader_election.cc:288] T 404e92da201544bd832722c66b5f39bc P a2cd23ce8ed84bf397ea32956ee82f1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ea1b3a2a7d24919a5a031949c2cb28b (127.0.98.2:36879), 0658d0cafe6c4e768d0b016696a5277e (127.0.98.3:42145)
I20260926 20:48:55.902765 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:55.898959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33709 (local address 127.0.98.2:36879)
0926 20:48:55.899006 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:55.899008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:55.899061 (+ 53us) server_negotiation.cc:407] Connection header received
0926 20:48:55.899130 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:55.899131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:55.899141 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:55.899211 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:48:55.899439 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.900216 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.901820 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.902376 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.902449 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:55.902459 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:55.902655 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 20:48:55.902694 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 20:48:55.902768 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:55.899055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37491 (local address 127.0.98.3:42145)
0926 20:48:55.899105 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:55.899107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:55.899115 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:48:55.899139 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:55.899140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:55.899150 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:55.899211 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:48:55.899430 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.900216 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.901821 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.902368 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.902449 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:55.902459 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:55.902654 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 20:48:55.902693 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 20:48:55.902812 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:55.898881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36879 (local address 127.0.98.1:33709)
0926 20:48:55.899002 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:55.899015 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:55.899071 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:55.899237 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:55.899238 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:55.899242 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:48:55.899359 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:55.899360 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.900305 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.900306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:55.901716 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:55.901717 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.902492 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.902493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:55.902568 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:55.902574 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 20:48:55.902631 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 20:48:55.902675 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":25,"thread_start_us":63,"threads_started":1}
I20260926 20:48:55.902812 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:55.898881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42145 (local address 127.0.98.1:37491)
0926 20:48:55.899004 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:55.899015 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:55.899067 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:55.899238 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:55.899239 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:55.899240 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:48:55.899354 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:55.899356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.900305 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.900306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:55.901699 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:55.901702 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.902491 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.902492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:55.902568 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:55.902572 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:48:55.902631 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 20:48:55.902675 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":52,"thread_start_us":59,"threads_started":1}
I20260926 20:48:55.903090 2421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "404e92da201544bd832722c66b5f39bc" candidate_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" is_pre_election: true
I20260926 20:48:55.903180 2421 raft_consensus.cc:2454] T 404e92da201544bd832722c66b5f39bc P 8ea1b3a2a7d24919a5a031949c2cb28b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2cd23ce8ed84bf397ea32956ee82f1a in term 0.
I20260926 20:48:55.903410 2215 leader_election.cc:302] T 404e92da201544bd832722c66b5f39bc P a2cd23ce8ed84bf397ea32956ee82f1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ea1b3a2a7d24919a5a031949c2cb28b, a2cd23ce8ed84bf397ea32956ee82f1a; no voters:
I20260926 20:48:55.903460 2549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "404e92da201544bd832722c66b5f39bc" candidate_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0658d0cafe6c4e768d0b016696a5277e" is_pre_election: true
I20260926 20:48:55.903563 2636 raft_consensus.cc:2781] T 404e92da201544bd832722c66b5f39bc P a2cd23ce8ed84bf397ea32956ee82f1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:48:55.903563 2549 raft_consensus.cc:2454] T 404e92da201544bd832722c66b5f39bc P 0658d0cafe6c4e768d0b016696a5277e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2cd23ce8ed84bf397ea32956ee82f1a in term 0.
I20260926 20:48:55.903600 2636 raft_consensus.cc:482] T 404e92da201544bd832722c66b5f39bc P a2cd23ce8ed84bf397ea32956ee82f1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:48:55.903631 2636 raft_consensus.cc:3037] T 404e92da201544bd832722c66b5f39bc P a2cd23ce8ed84bf397ea32956ee82f1a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:48:55.904392 2636 raft_consensus.cc:504] T 404e92da201544bd832722c66b5f39bc P a2cd23ce8ed84bf397ea32956ee82f1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } } peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } }
I20260926 20:48:55.904548 2636 leader_election.cc:288] T 404e92da201544bd832722c66b5f39bc P a2cd23ce8ed84bf397ea32956ee82f1a [CANDIDATE]: Term 1 election: Requested vote from peers 8ea1b3a2a7d24919a5a031949c2cb28b (127.0.98.2:36879), 0658d0cafe6c4e768d0b016696a5277e (127.0.98.3:42145)
I20260926 20:48:55.904704 2549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "404e92da201544bd832722c66b5f39bc" candidate_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0658d0cafe6c4e768d0b016696a5277e"
I20260926 20:48:55.904778 2549 raft_consensus.cc:3037] T 404e92da201544bd832722c66b5f39bc P 0658d0cafe6c4e768d0b016696a5277e [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:48:55.904771 2421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "404e92da201544bd832722c66b5f39bc" candidate_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b"
I20260926 20:48:55.904872 2421 raft_consensus.cc:3037] T 404e92da201544bd832722c66b5f39bc P 8ea1b3a2a7d24919a5a031949c2cb28b [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:48:55.905592 2549 raft_consensus.cc:2454] T 404e92da201544bd832722c66b5f39bc P 0658d0cafe6c4e768d0b016696a5277e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2cd23ce8ed84bf397ea32956ee82f1a in term 1.
I20260926 20:48:55.905668 2421 raft_consensus.cc:2454] T 404e92da201544bd832722c66b5f39bc P 8ea1b3a2a7d24919a5a031949c2cb28b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2cd23ce8ed84bf397ea32956ee82f1a in term 1.
I20260926 20:48:55.905817 2214 leader_election.cc:302] T 404e92da201544bd832722c66b5f39bc P a2cd23ce8ed84bf397ea32956ee82f1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0658d0cafe6c4e768d0b016696a5277e, a2cd23ce8ed84bf397ea32956ee82f1a; no voters:
I20260926 20:48:55.905934 2636 raft_consensus.cc:2781] T 404e92da201544bd832722c66b5f39bc P a2cd23ce8ed84bf397ea32956ee82f1a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:48:55.906097 2636 raft_consensus.cc:686] T 404e92da201544bd832722c66b5f39bc P a2cd23ce8ed84bf397ea32956ee82f1a [term 1 LEADER]: Becoming Leader. State: Replica: a2cd23ce8ed84bf397ea32956ee82f1a, State: Running, Role: LEADER
I20260926 20:48:55.906220 2636 consensus_queue.cc:237] T 404e92da201544bd832722c66b5f39bc P a2cd23ce8ed84bf397ea32956ee82f1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } } peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } }
I20260926 20:48:55.906853 2653 tablet_replica.cc:439] T 404e92da201544bd832722c66b5f39bc P a2cd23ce8ed84bf397ea32956ee82f1a: TxnStatusTablet state changed. Reason: New leader a2cd23ce8ed84bf397ea32956ee82f1a. Latest consensus state: current_term: 1 leader_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } } peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } } }
I20260926 20:48:55.906949 2653 tablet_replica.cc:442] T 404e92da201544bd832722c66b5f39bc P a2cd23ce8ed84bf397ea32956ee82f1a: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:48:55.907212 2655 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 20:48:55.907203 2124 catalog_manager.cc:5495] T 404e92da201544bd832722c66b5f39bc P a2cd23ce8ed84bf397ea32956ee82f1a reported cstate change: term changed from 0 to 1, leader changed from <none> to a2cd23ce8ed84bf397ea32956ee82f1a (127.0.98.1). New cstate: current_term: 1 leader_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } health_report { overall_health: UNKNOWN } } }
I20260926 20:48:55.907326 2655 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 20:48:55.924409 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:55.921630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44289 (local address 127.0.0.1:50824)
0926 20:48:55.921738 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:55.921752 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:55.921828 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:55.922078 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:55.922079 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:55.922081 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:48:55.922139 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:55.922142 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.922826 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.922828 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:55.923417 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:55.923419 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.923850 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.923851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:55.923920 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:55.923935 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:48:55.924244 (+ 309us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:48:55.924252 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 20:48:55.924294 (+ 42us) client_negotiation.cc:240] Negotiation successful
0926 20:48:55.924341 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 20:48:55.924402 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:55.921774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50824 (local address 127.0.98.1:44289)
0926 20:48:55.921845 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:55.921847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:55.921856 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:48:55.921912 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:55.921913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:55.921933 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:55.922071 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:48:55.922214 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.922733 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.923517 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.923733 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.923849 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:55.924061 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:48:55.924166 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:48:55.924257 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:55.924296 (+ 39us) server_negotiation.cc:299] Negotiation successful
0926 20:48:55.924321 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 20:48:55.926357 2421 raft_consensus.cc:1261] T 404e92da201544bd832722c66b5f39bc P 8ea1b3a2a7d24919a5a031949c2cb28b [term 1 FOLLOWER]: Refusing update from remote peer a2cd23ce8ed84bf397ea32956ee82f1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:48:55.926357 2549 raft_consensus.cc:1261] T 404e92da201544bd832722c66b5f39bc P 0658d0cafe6c4e768d0b016696a5277e [term 1 FOLLOWER]: Refusing update from remote peer a2cd23ce8ed84bf397ea32956ee82f1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:48:55.926689 2653 consensus_queue.cc:1035] T 404e92da201544bd832722c66b5f39bc P a2cd23ce8ed84bf397ea32956ee82f1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:48:55.926846 2636 consensus_queue.cc:1035] T 404e92da201544bd832722c66b5f39bc P a2cd23ce8ed84bf397ea32956ee82f1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:48:55.928049 2638 tablet_replica.cc:439] T 404e92da201544bd832722c66b5f39bc P 0658d0cafe6c4e768d0b016696a5277e: TxnStatusTablet state changed. Reason: New leader a2cd23ce8ed84bf397ea32956ee82f1a. Latest consensus state: current_term: 1 leader_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } } peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } } }
I20260926 20:48:55.928146 2638 tablet_replica.cc:442] T 404e92da201544bd832722c66b5f39bc P 0658d0cafe6c4e768d0b016696a5277e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:48:55.928090 2637 tablet_replica.cc:439] T 404e92da201544bd832722c66b5f39bc P 8ea1b3a2a7d24919a5a031949c2cb28b: TxnStatusTablet state changed. Reason: New leader a2cd23ce8ed84bf397ea32956ee82f1a. Latest consensus state: current_term: 1 leader_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } } peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } } }
I20260926 20:48:55.928175 2637 tablet_replica.cc:442] T 404e92da201544bd832722c66b5f39bc P 8ea1b3a2a7d24919a5a031949c2cb28b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:48:55.929409 2653 tablet_replica.cc:439] T 404e92da201544bd832722c66b5f39bc P a2cd23ce8ed84bf397ea32956ee82f1a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } } peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } } }
I20260926 20:48:55.929508 2653 tablet_replica.cc:442] T 404e92da201544bd832722c66b5f39bc P a2cd23ce8ed84bf397ea32956ee82f1a: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:48:55.929586 2659 mvcc.cc:205] Tried to move back new op lower bound from 7333706694350811136 to 7333706694272471040. Current Snapshot: MvccSnapshot[applied={T|T < 7333706694350811136}]
I20260926 20:48:55.929613 2653 tablet_replica.cc:439] T 404e92da201544bd832722c66b5f39bc P a2cd23ce8ed84bf397ea32956ee82f1a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } } peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } } }
I20260926 20:48:55.929669 2653 tablet_replica.cc:442] T 404e92da201544bd832722c66b5f39bc P a2cd23ce8ed84bf397ea32956ee82f1a: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:48:55.929677 2660 mvcc.cc:205] Tried to move back new op lower bound from 7333706694350811136 to 7333706694272471040. Current Snapshot: MvccSnapshot[applied={T|T < 7333706694350811136}]
I20260926 20:48:55.929657 2637 tablet_replica.cc:439] T 404e92da201544bd832722c66b5f39bc P 8ea1b3a2a7d24919a5a031949c2cb28b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } } peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } } }
I20260926 20:48:55.929746 2637 tablet_replica.cc:442] T 404e92da201544bd832722c66b5f39bc P 8ea1b3a2a7d24919a5a031949c2cb28b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:48:55.929813 2638 tablet_replica.cc:439] T 404e92da201544bd832722c66b5f39bc P 0658d0cafe6c4e768d0b016696a5277e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } } peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } } }
I20260926 20:48:55.929883 2638 tablet_replica.cc:442] T 404e92da201544bd832722c66b5f39bc P 0658d0cafe6c4e768d0b016696a5277e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:48:55.930040 2658 mvcc.cc:205] Tried to move back new op lower bound from 7333706694350811136 to 7333706694272471040. Current Snapshot: MvccSnapshot[applied={T|T < 7333706694350811136}]
I20260926 20:48:55.934808 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:55.932276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57512 (local address 127.0.98.3:42145)
0926 20:48:55.932425 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:55.932427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:55.932432 (+ 5us) server_negotiation.cc:407] Connection header received
0926 20:48:55.932455 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:55.932455 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:55.932469 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:55.932558 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:48:55.932661 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.933249 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.934046 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.934245 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.934326 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:55.934468 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:48:55.934568 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:48:55.934655 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:55.934703 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 20:48:55.934726 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20260926 20:48:55.940353 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:55.937555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42481 (local address 127.0.98.1:44289)
0926 20:48:55.937598 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:55.937600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:55.937675 (+ 75us) server_negotiation.cc:407] Connection header received
0926 20:48:55.937762 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:55.937762 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:55.937772 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:55.937815 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:48:55.937931 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.938374 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.939466 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.939697 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.939905 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:55.940121 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:48:55.940152 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:48:55.940205 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:55.940277 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 20:48:55.940295 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 20:48:55.940393 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:55.937488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44289 (local address 127.0.98.3:42481)
0926 20:48:55.937622 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:55.937636 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:55.937697 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:55.937818 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:55.937819 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:55.937820 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:48:55.937878 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:55.937879 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.938448 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.938448 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:55.939355 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:55.939357 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.939891 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.939893 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:55.939959 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:55.939976 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:48:55.940225 (+ 249us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:48:55.940227 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 20:48:55.940269 (+ 42us) client_negotiation.cc:240] Negotiation successful
0926 20:48:55.940312 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260926 20:48:55.942776 2421 raft_consensus.cc:1261] T dd515d26be474ab3960a9a3f039f20e4 P 8ea1b3a2a7d24919a5a031949c2cb28b [term 1 FOLLOWER]: Refusing update from remote peer 0658d0cafe6c4e768d0b016696a5277e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:48:55.942799 2281 raft_consensus.cc:1261] T dd515d26be474ab3960a9a3f039f20e4 P a2cd23ce8ed84bf397ea32956ee82f1a [term 1 FOLLOWER]: Refusing update from remote peer 0658d0cafe6c4e768d0b016696a5277e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:48:55.943044 2640 consensus_queue.cc:1035] T dd515d26be474ab3960a9a3f039f20e4 P 0658d0cafe6c4e768d0b016696a5277e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:48:55.943167 2638 consensus_queue.cc:1035] T dd515d26be474ab3960a9a3f039f20e4 P 0658d0cafe6c4e768d0b016696a5277e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:48:55.952431 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:55.949590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51535 (local address 127.0.98.3:42145)
0926 20:48:55.949631 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:55.949632 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:55.949641 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:48:55.949696 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:55.949697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:55.949709 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:55.949762 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:48:55.949934 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.950408 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.951598 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.951921 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.952013 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:55.952204 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:48:55.952239 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:48:55.952286 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:55.952333 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 20:48:55.952354 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 20:48:55.952464 2673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:55.949384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42145 (local address 127.0.98.1:51535)
0926 20:48:55.949567 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:55.949580 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:55.949634 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:55.949767 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:55.949768 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:55.949769 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:48:55.949850 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:55.949851 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.950497 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.950499 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:55.951505 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:55.951507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:55.952046 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:55.952047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:55.952104 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:55.952117 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:48:55.952285 (+ 168us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:48:55.952286 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 20:48:55.952333 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 20:48:55.952371 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":88,"threads_started":1}
I20260926 20:48:55.961680 2123 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:48:55.961807 2123 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:48:55.962615 2123 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ef56439e22a4401a988e847a088077e: Sending DeleteTablet for 3 replicas of tablet dd515d26be474ab3960a9a3f039f20e4
I20260926 20:48:55.962977 2541 tablet_service.cc:1514] Processing DeleteTablet for tablet dd515d26be474ab3960a9a3f039f20e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:48:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57502
I20260926 20:48:55.963016 2401 tablet_service.cc:1514] Processing DeleteTablet for tablet dd515d26be474ab3960a9a3f039f20e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:48:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58702
I20260926 20:48:55.963265 2255 tablet_service.cc:1514] Processing DeleteTablet for tablet dd515d26be474ab3960a9a3f039f20e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:48:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50812
I20260926 20:48:55.963490 2679 tablet_replica.cc:330] T dd515d26be474ab3960a9a3f039f20e4 P a2cd23ce8ed84bf397ea32956ee82f1a: stopping tablet replica
I20260926 20:48:55.963661 2679 raft_consensus.cc:2229] T dd515d26be474ab3960a9a3f039f20e4 P a2cd23ce8ed84bf397ea32956ee82f1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:48:55.963702 2123 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44652:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 20:48:55.963807 2679 raft_consensus.cc:2258] T dd515d26be474ab3960a9a3f039f20e4 P a2cd23ce8ed84bf397ea32956ee82f1a [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 20:48:55.963884 2123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:48:55.965088 2541 tablet_service.cc:1467] Processing CreateTablet for tablet dbeeb011a6c64fae973df3e405edd4a5 (DEFAULT_TABLE table=test-table [id=800ad0e351644750bab69a8f67842904]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:48:55.965157 2401 tablet_service.cc:1467] Processing CreateTablet for tablet dbeeb011a6c64fae973df3e405edd4a5 (DEFAULT_TABLE table=test-table [id=800ad0e351644750bab69a8f67842904]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:48:55.965227 2541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbeeb011a6c64fae973df3e405edd4a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:48:55.965279 2401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbeeb011a6c64fae973df3e405edd4a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:48:55.966709 2631 tablet_bootstrap.cc:492] T dbeeb011a6c64fae973df3e405edd4a5 P 8ea1b3a2a7d24919a5a031949c2cb28b: Bootstrap starting.
I20260926 20:48:55.966715 2255 tablet_service.cc:1467] Processing CreateTablet for tablet dbeeb011a6c64fae973df3e405edd4a5 (DEFAULT_TABLE table=test-table [id=800ad0e351644750bab69a8f67842904]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:48:55.966838 2255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbeeb011a6c64fae973df3e405edd4a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:48:55.967124 2631 tablet_bootstrap.cc:654] T dbeeb011a6c64fae973df3e405edd4a5 P 8ea1b3a2a7d24919a5a031949c2cb28b: Neither blocks nor log segments found. Creating new log.
I20260926 20:48:55.968146 2633 tablet_bootstrap.cc:492] T dbeeb011a6c64fae973df3e405edd4a5 P 0658d0cafe6c4e768d0b016696a5277e: Bootstrap starting.
I20260926 20:48:55.968148 2630 tablet_bootstrap.cc:492] T dbeeb011a6c64fae973df3e405edd4a5 P a2cd23ce8ed84bf397ea32956ee82f1a: Bootstrap starting.
I20260926 20:48:55.968567 2633 tablet_bootstrap.cc:654] T dbeeb011a6c64fae973df3e405edd4a5 P 0658d0cafe6c4e768d0b016696a5277e: Neither blocks nor log segments found. Creating new log.
I20260926 20:48:55.968568 2630 tablet_bootstrap.cc:654] T dbeeb011a6c64fae973df3e405edd4a5 P a2cd23ce8ed84bf397ea32956ee82f1a: Neither blocks nor log segments found. Creating new log.
I20260926 20:48:55.969378 2677 tablet_replica.cc:330] T dd515d26be474ab3960a9a3f039f20e4 P 0658d0cafe6c4e768d0b016696a5277e: stopping tablet replica
I20260926 20:48:55.969478 2679 ts_tablet_manager.cc:1905] T dd515d26be474ab3960a9a3f039f20e4 P a2cd23ce8ed84bf397ea32956ee82f1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:48:55.969501 2677 raft_consensus.cc:2229] T dd515d26be474ab3960a9a3f039f20e4 P 0658d0cafe6c4e768d0b016696a5277e [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:48:55.969611 2677 raft_consensus.cc:2258] T dd515d26be474ab3960a9a3f039f20e4 P 0658d0cafe6c4e768d0b016696a5277e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:48:55.970072 2633 tablet_bootstrap.cc:492] T dbeeb011a6c64fae973df3e405edd4a5 P 0658d0cafe6c4e768d0b016696a5277e: No bootstrap required, opened a new log
I20260926 20:48:55.970121 2633 ts_tablet_manager.cc:1397] T dbeeb011a6c64fae973df3e405edd4a5 P 0658d0cafe6c4e768d0b016696a5277e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:48:55.970224 2677 ts_tablet_manager.cc:1905] T dd515d26be474ab3960a9a3f039f20e4 P 0658d0cafe6c4e768d0b016696a5277e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:48:55.970291 2633 raft_consensus.cc:348] T dbeeb011a6c64fae973df3e405edd4a5 P 0658d0cafe6c4e768d0b016696a5277e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } } peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } }
I20260926 20:48:55.970382 2633 raft_consensus.cc:374] T dbeeb011a6c64fae973df3e405edd4a5 P 0658d0cafe6c4e768d0b016696a5277e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:48:55.970410 2633 raft_consensus.cc:729] T dbeeb011a6c64fae973df3e405edd4a5 P 0658d0cafe6c4e768d0b016696a5277e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0658d0cafe6c4e768d0b016696a5277e, State: Initialized, Role: FOLLOWER
I20260926 20:48:55.970470 2633 consensus_queue.cc:260] T dbeeb011a6c64fae973df3e405edd4a5 P 0658d0cafe6c4e768d0b016696a5277e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } } peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } }
I20260926 20:48:55.970590 2633 ts_tablet_manager.cc:1428] T dbeeb011a6c64fae973df3e405edd4a5 P 0658d0cafe6c4e768d0b016696a5277e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:48:55.971843 2679 ts_tablet_manager.cc:1918] T dd515d26be474ab3960a9a3f039f20e4 P a2cd23ce8ed84bf397ea32956ee82f1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:48:55.971921 2679 log.cc:1196] T dd515d26be474ab3960a9a3f039f20e4 P a2cd23ce8ed84bf397ea32956ee82f1a: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-0/wal/wals/dd515d26be474ab3960a9a3f039f20e4
I20260926 20:48:55.971976 2677 ts_tablet_manager.cc:1918] T dd515d26be474ab3960a9a3f039f20e4 P 0658d0cafe6c4e768d0b016696a5277e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:48:55.972051 2677 log.cc:1196] T dd515d26be474ab3960a9a3f039f20e4 P 0658d0cafe6c4e768d0b016696a5277e: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-2/wal/wals/dd515d26be474ab3960a9a3f039f20e4
I20260926 20:48:55.972306 2677 ts_tablet_manager.cc:1939] T dd515d26be474ab3960a9a3f039f20e4 P 0658d0cafe6c4e768d0b016696a5277e: Deleting consensus metadata
I20260926 20:48:55.972464 2678 tablet_replica.cc:330] T dd515d26be474ab3960a9a3f039f20e4 P 8ea1b3a2a7d24919a5a031949c2cb28b: stopping tablet replica
I20260926 20:48:55.972540 2679 ts_tablet_manager.cc:1939] T dd515d26be474ab3960a9a3f039f20e4 P a2cd23ce8ed84bf397ea32956ee82f1a: Deleting consensus metadata
I20260926 20:48:55.973009 2110 catalog_manager.cc:4841] TS 0658d0cafe6c4e768d0b016696a5277e (127.0.98.3:42145): tablet dd515d26be474ab3960a9a3f039f20e4 (table test-table [id=c87fb36613de426db1ff3bcd94d05218]) successfully deleted
I20260926 20:48:55.973124 2111 catalog_manager.cc:4841] TS a2cd23ce8ed84bf397ea32956ee82f1a (127.0.98.1:44289): tablet dd515d26be474ab3960a9a3f039f20e4 (table test-table [id=c87fb36613de426db1ff3bcd94d05218]) successfully deleted
I20260926 20:48:55.973194 2630 tablet_bootstrap.cc:492] T dbeeb011a6c64fae973df3e405edd4a5 P a2cd23ce8ed84bf397ea32956ee82f1a: No bootstrap required, opened a new log
I20260926 20:48:55.973248 2630 ts_tablet_manager.cc:1397] T dbeeb011a6c64fae973df3e405edd4a5 P a2cd23ce8ed84bf397ea32956ee82f1a: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260926 20:48:55.973359 2678 raft_consensus.cc:2229] T dd515d26be474ab3960a9a3f039f20e4 P 8ea1b3a2a7d24919a5a031949c2cb28b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:48:55.973385 2631 tablet_bootstrap.cc:492] T dbeeb011a6c64fae973df3e405edd4a5 P 8ea1b3a2a7d24919a5a031949c2cb28b: No bootstrap required, opened a new log
I20260926 20:48:55.973393 2630 raft_consensus.cc:348] T dbeeb011a6c64fae973df3e405edd4a5 P a2cd23ce8ed84bf397ea32956ee82f1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } } peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } }
I20260926 20:48:55.973471 2630 raft_consensus.cc:374] T dbeeb011a6c64fae973df3e405edd4a5 P a2cd23ce8ed84bf397ea32956ee82f1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:48:55.973487 2631 ts_tablet_manager.cc:1397] T dbeeb011a6c64fae973df3e405edd4a5 P 8ea1b3a2a7d24919a5a031949c2cb28b: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.000s
I20260926 20:48:55.973498 2630 raft_consensus.cc:729] T dbeeb011a6c64fae973df3e405edd4a5 P a2cd23ce8ed84bf397ea32956ee82f1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2cd23ce8ed84bf397ea32956ee82f1a, State: Initialized, Role: FOLLOWER
I20260926 20:48:55.973542 2630 consensus_queue.cc:260] T dbeeb011a6c64fae973df3e405edd4a5 P a2cd23ce8ed84bf397ea32956ee82f1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } } peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } }
I20260926 20:48:55.973582 2678 raft_consensus.cc:2258] T dd515d26be474ab3960a9a3f039f20e4 P 8ea1b3a2a7d24919a5a031949c2cb28b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:48:55.973649 2630 ts_tablet_manager.cc:1428] T dbeeb011a6c64fae973df3e405edd4a5 P a2cd23ce8ed84bf397ea32956ee82f1a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 20:48:55.973630 2631 raft_consensus.cc:348] T dbeeb011a6c64fae973df3e405edd4a5 P 8ea1b3a2a7d24919a5a031949c2cb28b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } } peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } }
I20260926 20:48:55.973753 2631 raft_consensus.cc:374] T dbeeb011a6c64fae973df3e405edd4a5 P 8ea1b3a2a7d24919a5a031949c2cb28b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:48:55.973778 2631 raft_consensus.cc:729] T dbeeb011a6c64fae973df3e405edd4a5 P 8ea1b3a2a7d24919a5a031949c2cb28b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ea1b3a2a7d24919a5a031949c2cb28b, State: Initialized, Role: FOLLOWER
I20260926 20:48:55.973819 2631 consensus_queue.cc:260] T dbeeb011a6c64fae973df3e405edd4a5 P 8ea1b3a2a7d24919a5a031949c2cb28b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } } peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } }
I20260926 20:48:55.973902 2631 ts_tablet_manager.cc:1428] T dbeeb011a6c64fae973df3e405edd4a5 P 8ea1b3a2a7d24919a5a031949c2cb28b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:48:55.974830 2678 ts_tablet_manager.cc:1905] T dd515d26be474ab3960a9a3f039f20e4 P 8ea1b3a2a7d24919a5a031949c2cb28b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:48:55.976531 2678 ts_tablet_manager.cc:1918] T dd515d26be474ab3960a9a3f039f20e4 P 8ea1b3a2a7d24919a5a031949c2cb28b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:48:55.976605 2678 log.cc:1196] T dd515d26be474ab3960a9a3f039f20e4 P 8ea1b3a2a7d24919a5a031949c2cb28b: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-1/wal/wals/dd515d26be474ab3960a9a3f039f20e4
I20260926 20:48:55.976850 2678 ts_tablet_manager.cc:1939] T dd515d26be474ab3960a9a3f039f20e4 P 8ea1b3a2a7d24919a5a031949c2cb28b: Deleting consensus metadata
I20260926 20:48:55.977404 2111 catalog_manager.cc:4841] TS 8ea1b3a2a7d24919a5a031949c2cb28b (127.0.98.2:36879): tablet dd515d26be474ab3960a9a3f039f20e4 (table test-table [id=c87fb36613de426db1ff3bcd94d05218]) successfully deleted
I20260926 20:48:56.006955 2675 raft_consensus.cc:482] T dbeeb011a6c64fae973df3e405edd4a5 P 0658d0cafe6c4e768d0b016696a5277e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:48:56.007081 2675 raft_consensus.cc:504] T dbeeb011a6c64fae973df3e405edd4a5 P 0658d0cafe6c4e768d0b016696a5277e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } } peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } }
I20260926 20:48:56.007253 2675 leader_election.cc:288] T dbeeb011a6c64fae973df3e405edd4a5 P 0658d0cafe6c4e768d0b016696a5277e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2cd23ce8ed84bf397ea32956ee82f1a (127.0.98.1:44289), 8ea1b3a2a7d24919a5a031949c2cb28b (127.0.98.2:36879)
I20260926 20:48:56.007450 2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbeeb011a6c64fae973df3e405edd4a5" candidate_uuid: "0658d0cafe6c4e768d0b016696a5277e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" is_pre_election: true
I20260926 20:48:56.007452 2421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbeeb011a6c64fae973df3e405edd4a5" candidate_uuid: "0658d0cafe6c4e768d0b016696a5277e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" is_pre_election: true
I20260926 20:48:56.007534 2281 raft_consensus.cc:2454] T dbeeb011a6c64fae973df3e405edd4a5 P a2cd23ce8ed84bf397ea32956ee82f1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0658d0cafe6c4e768d0b016696a5277e in term 0.
I20260926 20:48:56.007557 2421 raft_consensus.cc:2454] T dbeeb011a6c64fae973df3e405edd4a5 P 8ea1b3a2a7d24919a5a031949c2cb28b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0658d0cafe6c4e768d0b016696a5277e in term 0.
I20260926 20:48:56.007705 2495 leader_election.cc:302] T dbeeb011a6c64fae973df3e405edd4a5 P 0658d0cafe6c4e768d0b016696a5277e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0658d0cafe6c4e768d0b016696a5277e, 8ea1b3a2a7d24919a5a031949c2cb28b; no voters:
I20260926 20:48:56.007862 2675 raft_consensus.cc:2781] T dbeeb011a6c64fae973df3e405edd4a5 P 0658d0cafe6c4e768d0b016696a5277e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:48:56.007923 2675 raft_consensus.cc:482] T dbeeb011a6c64fae973df3e405edd4a5 P 0658d0cafe6c4e768d0b016696a5277e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:48:56.007944 2675 raft_consensus.cc:3037] T dbeeb011a6c64fae973df3e405edd4a5 P 0658d0cafe6c4e768d0b016696a5277e [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:48:56.008905 2675 raft_consensus.cc:504] T dbeeb011a6c64fae973df3e405edd4a5 P 0658d0cafe6c4e768d0b016696a5277e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } } peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } }
I20260926 20:48:56.009042 2675 leader_election.cc:288] T dbeeb011a6c64fae973df3e405edd4a5 P 0658d0cafe6c4e768d0b016696a5277e [CANDIDATE]: Term 1 election: Requested vote from peers a2cd23ce8ed84bf397ea32956ee82f1a (127.0.98.1:44289), 8ea1b3a2a7d24919a5a031949c2cb28b (127.0.98.2:36879)
I20260926 20:48:56.009262 2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbeeb011a6c64fae973df3e405edd4a5" candidate_uuid: "0658d0cafe6c4e768d0b016696a5277e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a"
I20260926 20:48:56.009342 2281 raft_consensus.cc:3037] T dbeeb011a6c64fae973df3e405edd4a5 P a2cd23ce8ed84bf397ea32956ee82f1a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:48:56.009289 2421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbeeb011a6c64fae973df3e405edd4a5" candidate_uuid: "0658d0cafe6c4e768d0b016696a5277e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b"
I20260926 20:48:56.009450 2421 raft_consensus.cc:3037] T dbeeb011a6c64fae973df3e405edd4a5 P 8ea1b3a2a7d24919a5a031949c2cb28b [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:48:56.010176 2281 raft_consensus.cc:2454] T dbeeb011a6c64fae973df3e405edd4a5 P a2cd23ce8ed84bf397ea32956ee82f1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0658d0cafe6c4e768d0b016696a5277e in term 1.
I20260926 20:48:56.010347 2495 leader_election.cc:302] T dbeeb011a6c64fae973df3e405edd4a5 P 0658d0cafe6c4e768d0b016696a5277e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0658d0cafe6c4e768d0b016696a5277e, a2cd23ce8ed84bf397ea32956ee82f1a; no voters:
I20260926 20:48:56.010421 2675 raft_consensus.cc:2781] T dbeeb011a6c64fae973df3e405edd4a5 P 0658d0cafe6c4e768d0b016696a5277e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:48:56.010456 2675 raft_consensus.cc:686] T dbeeb011a6c64fae973df3e405edd4a5 P 0658d0cafe6c4e768d0b016696a5277e [term 1 LEADER]: Becoming Leader. State: Replica: 0658d0cafe6c4e768d0b016696a5277e, State: Running, Role: LEADER
I20260926 20:48:56.010511 2675 consensus_queue.cc:237] T dbeeb011a6c64fae973df3e405edd4a5 P 0658d0cafe6c4e768d0b016696a5277e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } } peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } }
I20260926 20:48:56.010624 2421 raft_consensus.cc:2454] T dbeeb011a6c64fae973df3e405edd4a5 P 8ea1b3a2a7d24919a5a031949c2cb28b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0658d0cafe6c4e768d0b016696a5277e in term 1.
I20260926 20:48:56.011111 2123 catalog_manager.cc:5495] T dbeeb011a6c64fae973df3e405edd4a5 P 0658d0cafe6c4e768d0b016696a5277e reported cstate change: term changed from 0 to 1, leader changed from <none> to 0658d0cafe6c4e768d0b016696a5277e (127.0.98.3). New cstate: current_term: 1 leader_uuid: "0658d0cafe6c4e768d0b016696a5277e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } health_report { overall_health: HEALTHY } } }
I20260926 20:48:56.020581 2281 raft_consensus.cc:1261] T dbeeb011a6c64fae973df3e405edd4a5 P a2cd23ce8ed84bf397ea32956ee82f1a [term 1 FOLLOWER]: Refusing update from remote peer 0658d0cafe6c4e768d0b016696a5277e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:48:56.020789 2675 consensus_queue.cc:1035] T dbeeb011a6c64fae973df3e405edd4a5 P 0658d0cafe6c4e768d0b016696a5277e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:48:56.021046 2421 raft_consensus.cc:1261] T dbeeb011a6c64fae973df3e405edd4a5 P 8ea1b3a2a7d24919a5a031949c2cb28b [term 1 FOLLOWER]: Refusing update from remote peer 0658d0cafe6c4e768d0b016696a5277e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:48:56.021242 2675 consensus_queue.cc:1035] T dbeeb011a6c64fae973df3e405edd4a5 P 0658d0cafe6c4e768d0b016696a5277e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 20:48:56.026340 2660 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 20:48:57.545351 2124 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:48:57.545483 2124 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:48:57.546458 2124 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ef56439e22a4401a988e847a088077e: Sending DeleteTablet for 3 replicas of tablet dbeeb011a6c64fae973df3e405edd4a5
I20260926 20:48:57.546790 2401 tablet_service.cc:1514] Processing DeleteTablet for tablet dbeeb011a6c64fae973df3e405edd4a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:48:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58702
I20260926 20:48:57.547159 2709 tablet_replica.cc:330] T dbeeb011a6c64fae973df3e405edd4a5 P 8ea1b3a2a7d24919a5a031949c2cb28b: stopping tablet replica
I20260926 20:48:57.547154 2255 tablet_service.cc:1514] Processing DeleteTablet for tablet dbeeb011a6c64fae973df3e405edd4a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:48:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50812
I20260926 20:48:57.547214 2709 raft_consensus.cc:2229] T dbeeb011a6c64fae973df3e405edd4a5 P 8ea1b3a2a7d24919a5a031949c2cb28b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:48:57.547251 2709 raft_consensus.cc:2258] T dbeeb011a6c64fae973df3e405edd4a5 P 8ea1b3a2a7d24919a5a031949c2cb28b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:48:57.547334 2710 tablet_replica.cc:330] T dbeeb011a6c64fae973df3e405edd4a5 P a2cd23ce8ed84bf397ea32956ee82f1a: stopping tablet replica
I20260926 20:48:57.547381 2710 raft_consensus.cc:2229] T dbeeb011a6c64fae973df3e405edd4a5 P a2cd23ce8ed84bf397ea32956ee82f1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:48:57.547413 2710 raft_consensus.cc:2258] T dbeeb011a6c64fae973df3e405edd4a5 P a2cd23ce8ed84bf397ea32956ee82f1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:48:57.547524 2709 ts_tablet_manager.cc:1905] T dbeeb011a6c64fae973df3e405edd4a5 P 8ea1b3a2a7d24919a5a031949c2cb28b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:48:57.547622 2710 ts_tablet_manager.cc:1905] T dbeeb011a6c64fae973df3e405edd4a5 P a2cd23ce8ed84bf397ea32956ee82f1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:48:57.548938 2541 tablet_service.cc:1514] Processing DeleteTablet for tablet dbeeb011a6c64fae973df3e405edd4a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:48:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57502
I20260926 20:48:57.549316 2712 tablet_replica.cc:330] T dbeeb011a6c64fae973df3e405edd4a5 P 0658d0cafe6c4e768d0b016696a5277e: stopping tablet replica
I20260926 20:48:57.549422 2712 raft_consensus.cc:2229] T dbeeb011a6c64fae973df3e405edd4a5 P 0658d0cafe6c4e768d0b016696a5277e [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:48:57.549523 2712 raft_consensus.cc:2258] T dbeeb011a6c64fae973df3e405edd4a5 P 0658d0cafe6c4e768d0b016696a5277e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:48:57.549851 2709 ts_tablet_manager.cc:1918] T dbeeb011a6c64fae973df3e405edd4a5 P 8ea1b3a2a7d24919a5a031949c2cb28b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 20:48:57.549853 2710 ts_tablet_manager.cc:1918] T dbeeb011a6c64fae973df3e405edd4a5 P a2cd23ce8ed84bf397ea32956ee82f1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 20:48:57.549921 2710 log.cc:1196] T dbeeb011a6c64fae973df3e405edd4a5 P a2cd23ce8ed84bf397ea32956ee82f1a: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-0/wal/wals/dbeeb011a6c64fae973df3e405edd4a5
I20260926 20:48:57.549921 2709 log.cc:1196] T dbeeb011a6c64fae973df3e405edd4a5 P 8ea1b3a2a7d24919a5a031949c2cb28b: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-1/wal/wals/dbeeb011a6c64fae973df3e405edd4a5
I20260926 20:48:57.550165 2710 ts_tablet_manager.cc:1939] T dbeeb011a6c64fae973df3e405edd4a5 P a2cd23ce8ed84bf397ea32956ee82f1a: Deleting consensus metadata
I20260926 20:48:57.550180 2709 ts_tablet_manager.cc:1939] T dbeeb011a6c64fae973df3e405edd4a5 P 8ea1b3a2a7d24919a5a031949c2cb28b: Deleting consensus metadata
I20260926 20:48:57.550521 2111 catalog_manager.cc:4841] TS 8ea1b3a2a7d24919a5a031949c2cb28b (127.0.98.2:36879): tablet dbeeb011a6c64fae973df3e405edd4a5 (table test-table [id=800ad0e351644750bab69a8f67842904]) successfully deleted
I20260926 20:48:57.550606 2111 catalog_manager.cc:4841] TS a2cd23ce8ed84bf397ea32956ee82f1a (127.0.98.1:44289): tablet dbeeb011a6c64fae973df3e405edd4a5 (table test-table [id=800ad0e351644750bab69a8f67842904]) successfully deleted
I20260926 20:48:57.550885 2712 ts_tablet_manager.cc:1905] T dbeeb011a6c64fae973df3e405edd4a5 P 0658d0cafe6c4e768d0b016696a5277e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:48:57.552314 2712 ts_tablet_manager.cc:1918] T dbeeb011a6c64fae973df3e405edd4a5 P 0658d0cafe6c4e768d0b016696a5277e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 20:48:57.552361 2712 log.cc:1196] T dbeeb011a6c64fae973df3e405edd4a5 P 0658d0cafe6c4e768d0b016696a5277e: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-2/wal/wals/dbeeb011a6c64fae973df3e405edd4a5
I20260926 20:48:57.552577 2712 ts_tablet_manager.cc:1939] T dbeeb011a6c64fae973df3e405edd4a5 P 0658d0cafe6c4e768d0b016696a5277e: Deleting consensus metadata
I20260926 20:48:57.552908 2110 catalog_manager.cc:4841] TS 0658d0cafe6c4e768d0b016696a5277e (127.0.98.3:42145): tablet dbeeb011a6c64fae973df3e405edd4a5 (table test-table [id=800ad0e351644750bab69a8f67842904]) successfully deleted
Sep 26 20:48:57 dist-test-slave-hwlj krb5kdc[2079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455737, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 20:48:57 dist-test-slave-hwlj krb5kdc[2079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455737, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:48:57.578625 2719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:57.564116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44666 (local address 127.0.98.62:36087)
0926 20:48:57.564258 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:57.564260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:57.564279 (+ 19us) server_negotiation.cc:407] Connection header received
0926 20:48:57.564308 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:57.564309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:57.564319 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:57.564378 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:48:57.564926 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:57.565432 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:57.566257 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:57.566475 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:57.566542 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:57.575934 (+ 9392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:48:57.575936 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:48:57.575937 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:48:57.575945 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:48:57.577438 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:57.577796 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:57.577797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:57.577797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:57.577901 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:48:57.578139 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:48:57.578140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:48:57.578140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:48:57.578273 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:48:57.578349 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:57.578446 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 20:48:57.578513 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":78,"threads_started":1}
I20260926 20:48:57.580443 2124 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44666:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:48:57.580597 2124 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:48:57.582028 2255 tablet_service.cc:1467] Processing CreateTablet for tablet 6c5ae92fe41248d58fcdfc2894a6e972 (DEFAULT_TABLE table=test-table [id=393fd6b2afdb41d6a52beba864a32f1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:48:57.582062 2401 tablet_service.cc:1467] Processing CreateTablet for tablet 6c5ae92fe41248d58fcdfc2894a6e972 (DEFAULT_TABLE table=test-table [id=393fd6b2afdb41d6a52beba864a32f1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:48:57.582176 2255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c5ae92fe41248d58fcdfc2894a6e972. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:48:57.582201 2401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c5ae92fe41248d58fcdfc2894a6e972. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:48:57.582214 2541 tablet_service.cc:1467] Processing CreateTablet for tablet 6c5ae92fe41248d58fcdfc2894a6e972 (DEFAULT_TABLE table=test-table [id=393fd6b2afdb41d6a52beba864a32f1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:48:57.582316 2541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c5ae92fe41248d58fcdfc2894a6e972. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:48:57.583339 2720 tablet_bootstrap.cc:492] T 6c5ae92fe41248d58fcdfc2894a6e972 P a2cd23ce8ed84bf397ea32956ee82f1a: Bootstrap starting.
I20260926 20:48:57.583722 2721 tablet_bootstrap.cc:492] T 6c5ae92fe41248d58fcdfc2894a6e972 P 0658d0cafe6c4e768d0b016696a5277e: Bootstrap starting.
I20260926 20:48:57.583808 2722 tablet_bootstrap.cc:492] T 6c5ae92fe41248d58fcdfc2894a6e972 P 8ea1b3a2a7d24919a5a031949c2cb28b: Bootstrap starting.
I20260926 20:48:57.583823 2720 tablet_bootstrap.cc:654] T 6c5ae92fe41248d58fcdfc2894a6e972 P a2cd23ce8ed84bf397ea32956ee82f1a: Neither blocks nor log segments found. Creating new log.
I20260926 20:48:57.584125 2721 tablet_bootstrap.cc:654] T 6c5ae92fe41248d58fcdfc2894a6e972 P 0658d0cafe6c4e768d0b016696a5277e: Neither blocks nor log segments found. Creating new log.
I20260926 20:48:57.584143 2722 tablet_bootstrap.cc:654] T 6c5ae92fe41248d58fcdfc2894a6e972 P 8ea1b3a2a7d24919a5a031949c2cb28b: Neither blocks nor log segments found. Creating new log.
I20260926 20:48:57.584589 2720 tablet_bootstrap.cc:492] T 6c5ae92fe41248d58fcdfc2894a6e972 P a2cd23ce8ed84bf397ea32956ee82f1a: No bootstrap required, opened a new log
I20260926 20:48:57.584671 2720 ts_tablet_manager.cc:1397] T 6c5ae92fe41248d58fcdfc2894a6e972 P a2cd23ce8ed84bf397ea32956ee82f1a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 20:48:57.584784 2721 tablet_bootstrap.cc:492] T 6c5ae92fe41248d58fcdfc2894a6e972 P 0658d0cafe6c4e768d0b016696a5277e: No bootstrap required, opened a new log
I20260926 20:48:57.584836 2721 ts_tablet_manager.cc:1397] T 6c5ae92fe41248d58fcdfc2894a6e972 P 0658d0cafe6c4e768d0b016696a5277e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 20:48:57.584810 2720 raft_consensus.cc:348] T 6c5ae92fe41248d58fcdfc2894a6e972 P a2cd23ce8ed84bf397ea32956ee82f1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } } peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } }
I20260926 20:48:57.584887 2720 raft_consensus.cc:374] T 6c5ae92fe41248d58fcdfc2894a6e972 P a2cd23ce8ed84bf397ea32956ee82f1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:48:57.584908 2720 raft_consensus.cc:729] T 6c5ae92fe41248d58fcdfc2894a6e972 P a2cd23ce8ed84bf397ea32956ee82f1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2cd23ce8ed84bf397ea32956ee82f1a, State: Initialized, Role: FOLLOWER
I20260926 20:48:57.584934 2721 raft_consensus.cc:348] T 6c5ae92fe41248d58fcdfc2894a6e972 P 0658d0cafe6c4e768d0b016696a5277e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } } peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } }
I20260926 20:48:57.584956 2720 consensus_queue.cc:260] T 6c5ae92fe41248d58fcdfc2894a6e972 P a2cd23ce8ed84bf397ea32956ee82f1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } } peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } }
I20260926 20:48:57.585000 2721 raft_consensus.cc:374] T 6c5ae92fe41248d58fcdfc2894a6e972 P 0658d0cafe6c4e768d0b016696a5277e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:48:57.585021 2721 raft_consensus.cc:729] T 6c5ae92fe41248d58fcdfc2894a6e972 P 0658d0cafe6c4e768d0b016696a5277e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0658d0cafe6c4e768d0b016696a5277e, State: Initialized, Role: FOLLOWER
I20260926 20:48:57.585050 2720 ts_tablet_manager.cc:1428] T 6c5ae92fe41248d58fcdfc2894a6e972 P a2cd23ce8ed84bf397ea32956ee82f1a: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 20:48:57.585044 2722 tablet_bootstrap.cc:492] T 6c5ae92fe41248d58fcdfc2894a6e972 P 8ea1b3a2a7d24919a5a031949c2cb28b: No bootstrap required, opened a new log
I20260926 20:48:57.585094 2721 consensus_queue.cc:260] T 6c5ae92fe41248d58fcdfc2894a6e972 P 0658d0cafe6c4e768d0b016696a5277e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } } peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } }
I20260926 20:48:57.585199 2722 ts_tablet_manager.cc:1397] T 6c5ae92fe41248d58fcdfc2894a6e972 P 8ea1b3a2a7d24919a5a031949c2cb28b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:48:57.585215 2721 ts_tablet_manager.cc:1428] T 6c5ae92fe41248d58fcdfc2894a6e972 P 0658d0cafe6c4e768d0b016696a5277e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:48:57.585371 2722 raft_consensus.cc:348] T 6c5ae92fe41248d58fcdfc2894a6e972 P 8ea1b3a2a7d24919a5a031949c2cb28b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } } peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } }
I20260926 20:48:57.585448 2722 raft_consensus.cc:374] T 6c5ae92fe41248d58fcdfc2894a6e972 P 8ea1b3a2a7d24919a5a031949c2cb28b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:48:57.585470 2722 raft_consensus.cc:729] T 6c5ae92fe41248d58fcdfc2894a6e972 P 8ea1b3a2a7d24919a5a031949c2cb28b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ea1b3a2a7d24919a5a031949c2cb28b, State: Initialized, Role: FOLLOWER
I20260926 20:48:57.585517 2722 consensus_queue.cc:260] T 6c5ae92fe41248d58fcdfc2894a6e972 P 8ea1b3a2a7d24919a5a031949c2cb28b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } } peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } }
I20260926 20:48:57.585721 2722 ts_tablet_manager.cc:1428] T 6c5ae92fe41248d58fcdfc2894a6e972 P 8ea1b3a2a7d24919a5a031949c2cb28b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:48:57.592307 2703 raft_consensus.cc:482] T 6c5ae92fe41248d58fcdfc2894a6e972 P 0658d0cafe6c4e768d0b016696a5277e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:48:57.592389 2703 raft_consensus.cc:504] T 6c5ae92fe41248d58fcdfc2894a6e972 P 0658d0cafe6c4e768d0b016696a5277e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } } peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } }
I20260926 20:48:57.592518 2703 leader_election.cc:288] T 6c5ae92fe41248d58fcdfc2894a6e972 P 0658d0cafe6c4e768d0b016696a5277e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2cd23ce8ed84bf397ea32956ee82f1a (127.0.98.1:44289), 8ea1b3a2a7d24919a5a031949c2cb28b (127.0.98.2:36879)
I20260926 20:48:57.592711 2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c5ae92fe41248d58fcdfc2894a6e972" candidate_uuid: "0658d0cafe6c4e768d0b016696a5277e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" is_pre_election: true
I20260926 20:48:57.592803 2281 raft_consensus.cc:2454] T 6c5ae92fe41248d58fcdfc2894a6e972 P a2cd23ce8ed84bf397ea32956ee82f1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0658d0cafe6c4e768d0b016696a5277e in term 0.
I20260926 20:48:57.592769 2421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c5ae92fe41248d58fcdfc2894a6e972" candidate_uuid: "0658d0cafe6c4e768d0b016696a5277e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" is_pre_election: true
I20260926 20:48:57.592901 2421 raft_consensus.cc:2454] T 6c5ae92fe41248d58fcdfc2894a6e972 P 8ea1b3a2a7d24919a5a031949c2cb28b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0658d0cafe6c4e768d0b016696a5277e in term 0.
I20260926 20:48:57.592976 2495 leader_election.cc:302] T 6c5ae92fe41248d58fcdfc2894a6e972 P 0658d0cafe6c4e768d0b016696a5277e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0658d0cafe6c4e768d0b016696a5277e, a2cd23ce8ed84bf397ea32956ee82f1a; no voters:
I20260926 20:48:57.593092 2703 raft_consensus.cc:2781] T 6c5ae92fe41248d58fcdfc2894a6e972 P 0658d0cafe6c4e768d0b016696a5277e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:48:57.593127 2703 raft_consensus.cc:482] T 6c5ae92fe41248d58fcdfc2894a6e972 P 0658d0cafe6c4e768d0b016696a5277e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:48:57.593146 2703 raft_consensus.cc:3037] T 6c5ae92fe41248d58fcdfc2894a6e972 P 0658d0cafe6c4e768d0b016696a5277e [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:48:57.593808 2703 raft_consensus.cc:504] T 6c5ae92fe41248d58fcdfc2894a6e972 P 0658d0cafe6c4e768d0b016696a5277e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } } peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } }
I20260926 20:48:57.593950 2703 leader_election.cc:288] T 6c5ae92fe41248d58fcdfc2894a6e972 P 0658d0cafe6c4e768d0b016696a5277e [CANDIDATE]: Term 1 election: Requested vote from peers a2cd23ce8ed84bf397ea32956ee82f1a (127.0.98.1:44289), 8ea1b3a2a7d24919a5a031949c2cb28b (127.0.98.2:36879)
I20260926 20:48:57.594091 2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c5ae92fe41248d58fcdfc2894a6e972" candidate_uuid: "0658d0cafe6c4e768d0b016696a5277e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a"
I20260926 20:48:57.594132 2421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c5ae92fe41248d58fcdfc2894a6e972" candidate_uuid: "0658d0cafe6c4e768d0b016696a5277e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b"
I20260926 20:48:57.594178 2281 raft_consensus.cc:3037] T 6c5ae92fe41248d58fcdfc2894a6e972 P a2cd23ce8ed84bf397ea32956ee82f1a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:48:57.594190 2421 raft_consensus.cc:3037] T 6c5ae92fe41248d58fcdfc2894a6e972 P 8ea1b3a2a7d24919a5a031949c2cb28b [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:48:57.594899 2421 raft_consensus.cc:2454] T 6c5ae92fe41248d58fcdfc2894a6e972 P 8ea1b3a2a7d24919a5a031949c2cb28b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0658d0cafe6c4e768d0b016696a5277e in term 1.
I20260926 20:48:57.594899 2281 raft_consensus.cc:2454] T 6c5ae92fe41248d58fcdfc2894a6e972 P a2cd23ce8ed84bf397ea32956ee82f1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0658d0cafe6c4e768d0b016696a5277e in term 1.
I20260926 20:48:57.595095 2495 leader_election.cc:302] T 6c5ae92fe41248d58fcdfc2894a6e972 P 0658d0cafe6c4e768d0b016696a5277e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0658d0cafe6c4e768d0b016696a5277e, a2cd23ce8ed84bf397ea32956ee82f1a; no voters:
I20260926 20:48:57.595209 2703 raft_consensus.cc:2781] T 6c5ae92fe41248d58fcdfc2894a6e972 P 0658d0cafe6c4e768d0b016696a5277e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:48:57.595264 2703 raft_consensus.cc:686] T 6c5ae92fe41248d58fcdfc2894a6e972 P 0658d0cafe6c4e768d0b016696a5277e [term 1 LEADER]: Becoming Leader. State: Replica: 0658d0cafe6c4e768d0b016696a5277e, State: Running, Role: LEADER
I20260926 20:48:57.595332 2703 consensus_queue.cc:237] T 6c5ae92fe41248d58fcdfc2894a6e972 P 0658d0cafe6c4e768d0b016696a5277e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } } peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } }
I20260926 20:48:57.595880 2124 catalog_manager.cc:5495] T 6c5ae92fe41248d58fcdfc2894a6e972 P 0658d0cafe6c4e768d0b016696a5277e reported cstate change: term changed from 0 to 1, leader changed from <none> to 0658d0cafe6c4e768d0b016696a5277e (127.0.98.3). New cstate: current_term: 1 leader_uuid: "0658d0cafe6c4e768d0b016696a5277e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } health_report { overall_health: HEALTHY } } }
I20260926 20:48:57.605636 2729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:57.603262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57518 (local address 127.0.98.3:42145)
0926 20:48:57.603410 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:57.603412 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:57.603421 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:48:57.603443 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:57.603444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:57.603459 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:57.603522 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:48:57.603637 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:57.604211 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:57.604936 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:57.605157 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:57.605220 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:57.605338 (+ 118us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:48:57.605365 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:48:57.605404 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:57.605498 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 20:48:57.605527 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":103,"threads_started":1}
I20260926 20:48:57.607800 2421 raft_consensus.cc:1261] T 6c5ae92fe41248d58fcdfc2894a6e972 P 8ea1b3a2a7d24919a5a031949c2cb28b [term 1 FOLLOWER]: Refusing update from remote peer 0658d0cafe6c4e768d0b016696a5277e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:48:57.607800 2281 raft_consensus.cc:1261] T 6c5ae92fe41248d58fcdfc2894a6e972 P a2cd23ce8ed84bf397ea32956ee82f1a [term 1 FOLLOWER]: Refusing update from remote peer 0658d0cafe6c4e768d0b016696a5277e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:48:57.608011 2638 consensus_queue.cc:1035] T 6c5ae92fe41248d58fcdfc2894a6e972 P 0658d0cafe6c4e768d0b016696a5277e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:48:57.608094 2703 consensus_queue.cc:1035] T 6c5ae92fe41248d58fcdfc2894a6e972 P 0658d0cafe6c4e768d0b016696a5277e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:48:57.617766 2123 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:48:57.617864 2123 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:48:57.618541 2123 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ef56439e22a4401a988e847a088077e: Sending DeleteTablet for 3 replicas of tablet 6c5ae92fe41248d58fcdfc2894a6e972
I20260926 20:48:57.618848 2401 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c5ae92fe41248d58fcdfc2894a6e972 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:48:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58702
I20260926 20:48:57.618834 2541 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c5ae92fe41248d58fcdfc2894a6e972 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:48:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57502
I20260926 20:48:57.618887 2255 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c5ae92fe41248d58fcdfc2894a6e972 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:48:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50812
I20260926 20:48:57.618930 2709 tablet_replica.cc:330] T 6c5ae92fe41248d58fcdfc2894a6e972 P 8ea1b3a2a7d24919a5a031949c2cb28b: stopping tablet replica
I20260926 20:48:57.618939 2712 tablet_replica.cc:330] T 6c5ae92fe41248d58fcdfc2894a6e972 P 0658d0cafe6c4e768d0b016696a5277e: stopping tablet replica
I20260926 20:48:57.618971 2709 raft_consensus.cc:2229] T 6c5ae92fe41248d58fcdfc2894a6e972 P 8ea1b3a2a7d24919a5a031949c2cb28b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:48:57.619004 2712 raft_consensus.cc:2229] T 6c5ae92fe41248d58fcdfc2894a6e972 P 0658d0cafe6c4e768d0b016696a5277e [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:48:57.619009 2709 raft_consensus.cc:2258] T 6c5ae92fe41248d58fcdfc2894a6e972 P 8ea1b3a2a7d24919a5a031949c2cb28b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:48:57.619012 2710 tablet_replica.cc:330] T 6c5ae92fe41248d58fcdfc2894a6e972 P a2cd23ce8ed84bf397ea32956ee82f1a: stopping tablet replica
I20260926 20:48:57.619051 2712 raft_consensus.cc:2258] T 6c5ae92fe41248d58fcdfc2894a6e972 P 0658d0cafe6c4e768d0b016696a5277e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:48:57.619071 2710 raft_consensus.cc:2229] T 6c5ae92fe41248d58fcdfc2894a6e972 P a2cd23ce8ed84bf397ea32956ee82f1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:48:57.619115 2710 raft_consensus.cc:2258] T 6c5ae92fe41248d58fcdfc2894a6e972 P a2cd23ce8ed84bf397ea32956ee82f1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:48:57.619374 2712 ts_tablet_manager.cc:1905] T 6c5ae92fe41248d58fcdfc2894a6e972 P 0658d0cafe6c4e768d0b016696a5277e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:48:57.619374 2709 ts_tablet_manager.cc:1905] T 6c5ae92fe41248d58fcdfc2894a6e972 P 8ea1b3a2a7d24919a5a031949c2cb28b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:48:57.619416 2710 ts_tablet_manager.cc:1905] T 6c5ae92fe41248d58fcdfc2894a6e972 P a2cd23ce8ed84bf397ea32956ee82f1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:48:57.619534 2123 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44666:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:48:57.619668 2123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:48:57.621228 2709 ts_tablet_manager.cc:1918] T 6c5ae92fe41248d58fcdfc2894a6e972 P 8ea1b3a2a7d24919a5a031949c2cb28b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:48:57.621289 2709 log.cc:1196] T 6c5ae92fe41248d58fcdfc2894a6e972 P 8ea1b3a2a7d24919a5a031949c2cb28b: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-1/wal/wals/6c5ae92fe41248d58fcdfc2894a6e972
I20260926 20:48:57.621464 2710 ts_tablet_manager.cc:1918] T 6c5ae92fe41248d58fcdfc2894a6e972 P a2cd23ce8ed84bf397ea32956ee82f1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:48:57.621505 2709 ts_tablet_manager.cc:1939] T 6c5ae92fe41248d58fcdfc2894a6e972 P 8ea1b3a2a7d24919a5a031949c2cb28b: Deleting consensus metadata
I20260926 20:48:57.621517 2710 log.cc:1196] T 6c5ae92fe41248d58fcdfc2894a6e972 P a2cd23ce8ed84bf397ea32956ee82f1a: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-0/wal/wals/6c5ae92fe41248d58fcdfc2894a6e972
I20260926 20:48:57.621711 2710 ts_tablet_manager.cc:1939] T 6c5ae92fe41248d58fcdfc2894a6e972 P a2cd23ce8ed84bf397ea32956ee82f1a: Deleting consensus metadata
I20260926 20:48:57.621731 2255 tablet_service.cc:1467] Processing CreateTablet for tablet 07432dfc4f164cecb26d120019b2ec58 (DEFAULT_TABLE table=test-table [id=61cf043453f44ffebaed02ab5cbd69f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:48:57.621798 2541 tablet_service.cc:1467] Processing CreateTablet for tablet 07432dfc4f164cecb26d120019b2ec58 (DEFAULT_TABLE table=test-table [id=61cf043453f44ffebaed02ab5cbd69f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:48:57.621840 2255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07432dfc4f164cecb26d120019b2ec58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:48:57.622005 2111 catalog_manager.cc:4841] TS a2cd23ce8ed84bf397ea32956ee82f1a (127.0.98.1:44289): tablet 6c5ae92fe41248d58fcdfc2894a6e972 (table test-table [id=393fd6b2afdb41d6a52beba864a32f1a]) successfully deleted
I20260926 20:48:57.622066 2111 catalog_manager.cc:4841] TS 8ea1b3a2a7d24919a5a031949c2cb28b (127.0.98.2:36879): tablet 6c5ae92fe41248d58fcdfc2894a6e972 (table test-table [id=393fd6b2afdb41d6a52beba864a32f1a]) successfully deleted
I20260926 20:48:57.622077 2401 tablet_service.cc:1467] Processing CreateTablet for tablet 07432dfc4f164cecb26d120019b2ec58 (DEFAULT_TABLE table=test-table [id=61cf043453f44ffebaed02ab5cbd69f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:48:57.622164 2401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07432dfc4f164cecb26d120019b2ec58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:48:57.622252 2541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07432dfc4f164cecb26d120019b2ec58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:48:57.623062 2720 tablet_bootstrap.cc:492] T 07432dfc4f164cecb26d120019b2ec58 P a2cd23ce8ed84bf397ea32956ee82f1a: Bootstrap starting.
I20260926 20:48:57.623348 2721 tablet_bootstrap.cc:492] T 07432dfc4f164cecb26d120019b2ec58 P 0658d0cafe6c4e768d0b016696a5277e: Bootstrap starting.
I20260926 20:48:57.623410 2720 tablet_bootstrap.cc:654] T 07432dfc4f164cecb26d120019b2ec58 P a2cd23ce8ed84bf397ea32956ee82f1a: Neither blocks nor log segments found. Creating new log.
I20260926 20:48:57.622252 2712 ts_tablet_manager.cc:1918] T 6c5ae92fe41248d58fcdfc2894a6e972 P 0658d0cafe6c4e768d0b016696a5277e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:48:57.623466 2712 log.cc:1196] T 6c5ae92fe41248d58fcdfc2894a6e972 P 0658d0cafe6c4e768d0b016696a5277e: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-2/wal/wals/6c5ae92fe41248d58fcdfc2894a6e972
I20260926 20:48:57.623651 2721 tablet_bootstrap.cc:654] T 07432dfc4f164cecb26d120019b2ec58 P 0658d0cafe6c4e768d0b016696a5277e: Neither blocks nor log segments found. Creating new log.
I20260926 20:48:57.623683 2712 ts_tablet_manager.cc:1939] T 6c5ae92fe41248d58fcdfc2894a6e972 P 0658d0cafe6c4e768d0b016696a5277e: Deleting consensus metadata
I20260926 20:48:57.623910 2722 tablet_bootstrap.cc:492] T 07432dfc4f164cecb26d120019b2ec58 P 8ea1b3a2a7d24919a5a031949c2cb28b: Bootstrap starting.
I20260926 20:48:57.623992 2720 tablet_bootstrap.cc:492] T 07432dfc4f164cecb26d120019b2ec58 P a2cd23ce8ed84bf397ea32956ee82f1a: No bootstrap required, opened a new log
I20260926 20:48:57.624001 2110 catalog_manager.cc:4841] TS 0658d0cafe6c4e768d0b016696a5277e (127.0.98.3:42145): tablet 6c5ae92fe41248d58fcdfc2894a6e972 (table test-table [id=393fd6b2afdb41d6a52beba864a32f1a]) successfully deleted
I20260926 20:48:57.624043 2720 ts_tablet_manager.cc:1397] T 07432dfc4f164cecb26d120019b2ec58 P a2cd23ce8ed84bf397ea32956ee82f1a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 20:48:57.624203 2720 raft_consensus.cc:348] T 07432dfc4f164cecb26d120019b2ec58 P a2cd23ce8ed84bf397ea32956ee82f1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } } peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } }
I20260926 20:48:57.624289 2720 raft_consensus.cc:374] T 07432dfc4f164cecb26d120019b2ec58 P a2cd23ce8ed84bf397ea32956ee82f1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:48:57.624315 2720 raft_consensus.cc:729] T 07432dfc4f164cecb26d120019b2ec58 P a2cd23ce8ed84bf397ea32956ee82f1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2cd23ce8ed84bf397ea32956ee82f1a, State: Initialized, Role: FOLLOWER
I20260926 20:48:57.624352 2722 tablet_bootstrap.cc:654] T 07432dfc4f164cecb26d120019b2ec58 P 8ea1b3a2a7d24919a5a031949c2cb28b: Neither blocks nor log segments found. Creating new log.
I20260926 20:48:57.624363 2720 consensus_queue.cc:260] T 07432dfc4f164cecb26d120019b2ec58 P a2cd23ce8ed84bf397ea32956ee82f1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } } peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } }
I20260926 20:48:57.624475 2720 ts_tablet_manager.cc:1428] T 07432dfc4f164cecb26d120019b2ec58 P a2cd23ce8ed84bf397ea32956ee82f1a: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 20:48:57.625324 2722 tablet_bootstrap.cc:492] T 07432dfc4f164cecb26d120019b2ec58 P 8ea1b3a2a7d24919a5a031949c2cb28b: No bootstrap required, opened a new log
I20260926 20:48:57.625319 2721 tablet_bootstrap.cc:492] T 07432dfc4f164cecb26d120019b2ec58 P 0658d0cafe6c4e768d0b016696a5277e: No bootstrap required, opened a new log
I20260926 20:48:57.625386 2721 ts_tablet_manager.cc:1397] T 07432dfc4f164cecb26d120019b2ec58 P 0658d0cafe6c4e768d0b016696a5277e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 20:48:57.625371 2722 ts_tablet_manager.cc:1397] T 07432dfc4f164cecb26d120019b2ec58 P 8ea1b3a2a7d24919a5a031949c2cb28b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:48:57.625514 2721 raft_consensus.cc:348] T 07432dfc4f164cecb26d120019b2ec58 P 0658d0cafe6c4e768d0b016696a5277e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } } peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } }
I20260926 20:48:57.625526 2722 raft_consensus.cc:348] T 07432dfc4f164cecb26d120019b2ec58 P 8ea1b3a2a7d24919a5a031949c2cb28b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } } peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } }
I20260926 20:48:57.625589 2721 raft_consensus.cc:374] T 07432dfc4f164cecb26d120019b2ec58 P 0658d0cafe6c4e768d0b016696a5277e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:48:57.625617 2721 raft_consensus.cc:729] T 07432dfc4f164cecb26d120019b2ec58 P 0658d0cafe6c4e768d0b016696a5277e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0658d0cafe6c4e768d0b016696a5277e, State: Initialized, Role: FOLLOWER
I20260926 20:48:57.625594 2722 raft_consensus.cc:374] T 07432dfc4f164cecb26d120019b2ec58 P 8ea1b3a2a7d24919a5a031949c2cb28b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:48:57.625644 2722 raft_consensus.cc:729] T 07432dfc4f164cecb26d120019b2ec58 P 8ea1b3a2a7d24919a5a031949c2cb28b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ea1b3a2a7d24919a5a031949c2cb28b, State: Initialized, Role: FOLLOWER
I20260926 20:48:57.625664 2721 consensus_queue.cc:260] T 07432dfc4f164cecb26d120019b2ec58 P 0658d0cafe6c4e768d0b016696a5277e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } } peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } }
I20260926 20:48:57.625685 2722 consensus_queue.cc:260] T 07432dfc4f164cecb26d120019b2ec58 P 8ea1b3a2a7d24919a5a031949c2cb28b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } } peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } }
I20260926 20:48:57.625767 2721 ts_tablet_manager.cc:1428] T 07432dfc4f164cecb26d120019b2ec58 P 0658d0cafe6c4e768d0b016696a5277e: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 20:48:57.625784 2722 ts_tablet_manager.cc:1428] T 07432dfc4f164cecb26d120019b2ec58 P 8ea1b3a2a7d24919a5a031949c2cb28b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 20:48:57.715243 2726 raft_consensus.cc:482] T 07432dfc4f164cecb26d120019b2ec58 P 8ea1b3a2a7d24919a5a031949c2cb28b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:48:57.715364 2726 raft_consensus.cc:504] T 07432dfc4f164cecb26d120019b2ec58 P 8ea1b3a2a7d24919a5a031949c2cb28b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } } peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } }
I20260926 20:48:57.715775 2726 leader_election.cc:288] T 07432dfc4f164cecb26d120019b2ec58 P 8ea1b3a2a7d24919a5a031949c2cb28b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2cd23ce8ed84bf397ea32956ee82f1a (127.0.98.1:44289), 0658d0cafe6c4e768d0b016696a5277e (127.0.98.3:42145)
I20260926 20:48:57.720011 2729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:57.715990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42501 (local address 127.0.98.3:42145)
0926 20:48:57.716036 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:57.716038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:57.716247 (+ 209us) server_negotiation.cc:407] Connection header received
0926 20:48:57.716278 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:57.716279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:57.716290 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:57.716341 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:48:57.716601 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:57.717362 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:57.718991 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:57.719549 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:57.719626 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:57.719638 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:57.719896 (+ 258us) server_negotiation.cc:299] Negotiation successful
0926 20:48:57.719937 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 20:48:57.720012 2738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:57.715996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60867 (local address 127.0.98.1:44289)
0926 20:48:57.716109 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:57.716110 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:57.716188 (+ 78us) server_negotiation.cc:407] Connection header received
0926 20:48:57.716237 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:57.716238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:57.716250 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:57.716335 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:48:57.716602 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:57.717344 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:57.718990 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:57.719549 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:57.719626 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:57.719638 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:57.719896 (+ 258us) server_negotiation.cc:299] Negotiation successful
0926 20:48:57.719937 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":55,"threads_started":1}
I20260926 20:48:57.720067 2737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:57.715936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44289 (local address 127.0.98.2:60867)
0926 20:48:57.716102 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:57.716114 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:57.716181 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:57.716399 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:57.716401 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:57.716405 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:48:57.716513 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:57.716514 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:57.717504 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:57.717505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:57.718883 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:57.718886 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:57.719673 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:57.719674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:57.719802 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:57.719805 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:48:57.719878 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 20:48:57.719943 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":117,"thread_start_us":121,"threads_started":1}
I20260926 20:48:57.720359 2736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:57.715943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42145 (local address 127.0.98.2:42501)
0926 20:48:57.716107 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:57.716116 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:57.716181 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:57.716399 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:57.716401 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:57.716403 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:48:57.716512 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:57.716514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:57.717458 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:57.717459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:57.718883 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:57.718886 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:57.719673 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:57.719674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:57.719800 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:57.719808 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 20:48:57.719878 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 20:48:57.719943 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":81,"thread_start_us":76,"threads_started":1}
I20260926 20:48:57.720446 2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07432dfc4f164cecb26d120019b2ec58" candidate_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" is_pre_election: true
I20260926 20:48:57.720556 2281 raft_consensus.cc:2454] T 07432dfc4f164cecb26d120019b2ec58 P a2cd23ce8ed84bf397ea32956ee82f1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ea1b3a2a7d24919a5a031949c2cb28b in term 0.
I20260926 20:48:57.720672 2549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07432dfc4f164cecb26d120019b2ec58" candidate_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0658d0cafe6c4e768d0b016696a5277e" is_pre_election: true
I20260926 20:48:57.720741 2355 leader_election.cc:302] T 07432dfc4f164cecb26d120019b2ec58 P 8ea1b3a2a7d24919a5a031949c2cb28b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ea1b3a2a7d24919a5a031949c2cb28b, a2cd23ce8ed84bf397ea32956ee82f1a; no voters:
I20260926 20:48:57.720779 2549 raft_consensus.cc:2454] T 07432dfc4f164cecb26d120019b2ec58 P 0658d0cafe6c4e768d0b016696a5277e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ea1b3a2a7d24919a5a031949c2cb28b in term 0.
I20260926 20:48:57.720858 2726 raft_consensus.cc:2781] T 07432dfc4f164cecb26d120019b2ec58 P 8ea1b3a2a7d24919a5a031949c2cb28b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:48:57.720901 2726 raft_consensus.cc:482] T 07432dfc4f164cecb26d120019b2ec58 P 8ea1b3a2a7d24919a5a031949c2cb28b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:48:57.720925 2726 raft_consensus.cc:3037] T 07432dfc4f164cecb26d120019b2ec58 P 8ea1b3a2a7d24919a5a031949c2cb28b [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:48:57.721640 2726 raft_consensus.cc:504] T 07432dfc4f164cecb26d120019b2ec58 P 8ea1b3a2a7d24919a5a031949c2cb28b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } } peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } }
I20260926 20:48:57.721783 2726 leader_election.cc:288] T 07432dfc4f164cecb26d120019b2ec58 P 8ea1b3a2a7d24919a5a031949c2cb28b [CANDIDATE]: Term 1 election: Requested vote from peers a2cd23ce8ed84bf397ea32956ee82f1a (127.0.98.1:44289), 0658d0cafe6c4e768d0b016696a5277e (127.0.98.3:42145)
I20260926 20:48:57.721925 2549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07432dfc4f164cecb26d120019b2ec58" candidate_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0658d0cafe6c4e768d0b016696a5277e"
I20260926 20:48:57.721995 2549 raft_consensus.cc:3037] T 07432dfc4f164cecb26d120019b2ec58 P 0658d0cafe6c4e768d0b016696a5277e [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:48:57.721951 2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07432dfc4f164cecb26d120019b2ec58" candidate_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a"
I20260926 20:48:57.722100 2281 raft_consensus.cc:3037] T 07432dfc4f164cecb26d120019b2ec58 P a2cd23ce8ed84bf397ea32956ee82f1a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:48:57.722640 2549 raft_consensus.cc:2454] T 07432dfc4f164cecb26d120019b2ec58 P 0658d0cafe6c4e768d0b016696a5277e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ea1b3a2a7d24919a5a031949c2cb28b in term 1.
I20260926 20:48:57.722811 2354 leader_election.cc:302] T 07432dfc4f164cecb26d120019b2ec58 P 8ea1b3a2a7d24919a5a031949c2cb28b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0658d0cafe6c4e768d0b016696a5277e, 8ea1b3a2a7d24919a5a031949c2cb28b; no voters:
I20260926 20:48:57.722926 2726 raft_consensus.cc:2781] T 07432dfc4f164cecb26d120019b2ec58 P 8ea1b3a2a7d24919a5a031949c2cb28b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:48:57.723088 2726 raft_consensus.cc:686] T 07432dfc4f164cecb26d120019b2ec58 P 8ea1b3a2a7d24919a5a031949c2cb28b [term 1 LEADER]: Becoming Leader. State: Replica: 8ea1b3a2a7d24919a5a031949c2cb28b, State: Running, Role: LEADER
I20260926 20:48:57.723086 2281 raft_consensus.cc:2454] T 07432dfc4f164cecb26d120019b2ec58 P a2cd23ce8ed84bf397ea32956ee82f1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ea1b3a2a7d24919a5a031949c2cb28b in term 1.
I20260926 20:48:57.723167 2726 consensus_queue.cc:237] T 07432dfc4f164cecb26d120019b2ec58 P 8ea1b3a2a7d24919a5a031949c2cb28b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } } peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } }
I20260926 20:48:57.723794 2124 catalog_manager.cc:5495] T 07432dfc4f164cecb26d120019b2ec58 P 8ea1b3a2a7d24919a5a031949c2cb28b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ea1b3a2a7d24919a5a031949c2cb28b (127.0.98.2). New cstate: current_term: 1 leader_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ea1b3a2a7d24919a5a031949c2cb28b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 } health_report { overall_health: UNKNOWN } } }
I20260926 20:48:57.745467 2741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:57.742701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58714 (local address 127.0.98.2:36879)
0926 20:48:57.742880 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:57.742882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:57.742890 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:48:57.742930 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:57.742931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:57.742958 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:57.743030 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:48:57.743207 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:57.743711 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:57.744645 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:57.744852 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:57.744908 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:57.745099 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:48:57.745222 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:48:57.745288 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:57.745375 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 20:48:57.745392 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":100,"threads_started":1}
I20260926 20:48:57.749982 2738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:57.747910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52101 (local address 127.0.98.1:44289)
0926 20:48:57.747968 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:57.747970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:57.748055 (+ 85us) server_negotiation.cc:407] Connection header received
0926 20:48:57.748083 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:57.748083 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:57.748093 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:57.748138 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:48:57.748264 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:57.748722 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:57.749420 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:57.749593 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:57.749640 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:57.749795 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:48:57.749819 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:48:57.749856 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:57.749908 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 20:48:57.749925 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 20:48:57.750015 2744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:57.747827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44289 (local address 127.0.98.2:52101)
0926 20:48:57.747981 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:57.747991 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:57.748049 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:57.748158 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:57.748159 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:57.748160 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:48:57.748209 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:57.748210 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:57.748791 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:57.748792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:57.749351 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:57.749352 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:57.749672 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:57.749673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:57.749715 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:57.749727 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:48:57.749872 (+ 145us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:48:57.749873 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 20:48:57.749907 (+ 34us) client_negotiation.cc:240] Negotiation successful
0926 20:48:57.749938 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":83,"threads_started":1}
I20260926 20:48:57.752352 2549 raft_consensus.cc:1261] T 07432dfc4f164cecb26d120019b2ec58 P 0658d0cafe6c4e768d0b016696a5277e [term 1 FOLLOWER]: Refusing update from remote peer 8ea1b3a2a7d24919a5a031949c2cb28b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:48:57.752373 2281 raft_consensus.cc:1261] T 07432dfc4f164cecb26d120019b2ec58 P a2cd23ce8ed84bf397ea32956ee82f1a [term 1 FOLLOWER]: Refusing update from remote peer 8ea1b3a2a7d24919a5a031949c2cb28b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:48:57.752607 2726 consensus_queue.cc:1035] T 07432dfc4f164cecb26d120019b2ec58 P 8ea1b3a2a7d24919a5a031949c2cb28b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0658d0cafe6c4e768d0b016696a5277e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:48:57.752758 2739 consensus_queue.cc:1035] T 07432dfc4f164cecb26d120019b2ec58 P 8ea1b3a2a7d24919a5a031949c2cb28b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2cd23ce8ed84bf397ea32956ee82f1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:48:57.759430 2741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:57.757021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56011 (local address 127.0.98.2:36879)
0926 20:48:57.757107 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 20:48:57.757110 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:48:57.757164 (+ 54us) server_negotiation.cc:407] Connection header received
0926 20:48:57.757243 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:48:57.757243 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:48:57.757254 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:48:57.757306 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:48:57.757452 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:57.757958 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:57.758670 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:48:57.758974 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:48:57.759022 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:57.759203 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:48:57.759237 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:48:57.759284 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 20:48:57.759343 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 20:48:57.759364 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 20:48:57.759464 2749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:57.756985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36879 (local address 127.0.98.1:56011)
0926 20:48:57.757107 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:57.757118 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:48:57.757189 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:48:57.757328 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:48:57.757329 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:48:57.757331 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:48:57.757383 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:57.757385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:57.758030 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:57.758031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:57.758578 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:48:57.758579 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:48:57.759033 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:48:57.759034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:48:57.759078 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:48:57.759082 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:48:57.759297 (+ 215us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:48:57.759300 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:48:57.759342 (+ 42us) client_negotiation.cc:240] Negotiation successful
0926 20:48:57.759386 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":79,"threads_started":1}
W20260926 20:48:57.759804 2692 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 20:48:59.277868 2124 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:48:59.278012 2124 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:48:59.279089 2124 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ef56439e22a4401a988e847a088077e: Sending DeleteTablet for 3 replicas of tablet 07432dfc4f164cecb26d120019b2ec58
I20260926 20:48:59.279311 2255 tablet_service.cc:1514] Processing DeleteTablet for tablet 07432dfc4f164cecb26d120019b2ec58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:48:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50812
I20260926 20:48:59.279378 2541 tablet_service.cc:1514] Processing DeleteTablet for tablet 07432dfc4f164cecb26d120019b2ec58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:48:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57502
I20260926 20:48:59.279374 2401 tablet_service.cc:1514] Processing DeleteTablet for tablet 07432dfc4f164cecb26d120019b2ec58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:48:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58702
I20260926 20:48:59.279520 2771 tablet_replica.cc:330] T 07432dfc4f164cecb26d120019b2ec58 P a2cd23ce8ed84bf397ea32956ee82f1a: stopping tablet replica
I20260926 20:48:59.279582 2771 raft_consensus.cc:2229] T 07432dfc4f164cecb26d120019b2ec58 P a2cd23ce8ed84bf397ea32956ee82f1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:48:59.279618 2771 raft_consensus.cc:2258] T 07432dfc4f164cecb26d120019b2ec58 P a2cd23ce8ed84bf397ea32956ee82f1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:48:59.279670 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 2196
W20260926 20:48:59.288895 2111 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44289 (error 108)
I20260926 20:48:59.289084 2772 tablet_replica.cc:330] T 07432dfc4f164cecb26d120019b2ec58 P 0658d0cafe6c4e768d0b016696a5277e: stopping tablet replica
I20260926 20:48:59.289182 2772 raft_consensus.cc:2229] T 07432dfc4f164cecb26d120019b2ec58 P 0658d0cafe6c4e768d0b016696a5277e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:48:59.289242 2772 raft_consensus.cc:2258] T 07432dfc4f164cecb26d120019b2ec58 P 0658d0cafe6c4e768d0b016696a5277e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:48:59.289389 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 2336
I20260926 20:48:59.289397 2773 tablet_replica.cc:330] T 07432dfc4f164cecb26d120019b2ec58 P 8ea1b3a2a7d24919a5a031949c2cb28b: stopping tablet replica
I20260926 20:48:59.289454 2773 raft_consensus.cc:2229] T 07432dfc4f164cecb26d120019b2ec58 P 8ea1b3a2a7d24919a5a031949c2cb28b [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:48:59.289556 2773 raft_consensus.cc:2258] T 07432dfc4f164cecb26d120019b2ec58 P 8ea1b3a2a7d24919a5a031949c2cb28b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:48:59.289610 2772 ts_tablet_manager.cc:1905] T 07432dfc4f164cecb26d120019b2ec58 P 0658d0cafe6c4e768d0b016696a5277e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:48:59.291115 2775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:59.289295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44289 (local address 127.0.0.1:48286)
0926 20:48:59.291030 (+ 1735us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:59.291080 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1710,"thread_start_us":65,"threads_started":1}
W20260926 20:48:59.291239 2111 catalog_manager.cc:4584] TS a2cd23ce8ed84bf397ea32956ee82f1a (127.0.98.1:44289): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 07432dfc4f164cecb26d120019b2ec58: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44289: connect: Connection refused (error 111)
I20260926 20:48:59.291452 2772 ts_tablet_manager.cc:1918] T 07432dfc4f164cecb26d120019b2ec58 P 0658d0cafe6c4e768d0b016696a5277e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 20:48:59.291492 2772 log.cc:1196] T 07432dfc4f164cecb26d120019b2ec58 P 0658d0cafe6c4e768d0b016696a5277e: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455725523895-392-0/minicluster-data/ts-2/wal/wals/07432dfc4f164cecb26d120019b2ec58
I20260926 20:48:59.291663 2772 ts_tablet_manager.cc:1939] T 07432dfc4f164cecb26d120019b2ec58 P 0658d0cafe6c4e768d0b016696a5277e: Deleting consensus metadata
I20260926 20:48:59.292029 2110 catalog_manager.cc:4841] TS 0658d0cafe6c4e768d0b016696a5277e (127.0.98.3:42145): tablet 07432dfc4f164cecb26d120019b2ec58 (table test-table [id=61cf043453f44ffebaed02ab5cbd69f4]) successfully deleted
I20260926 20:48:59.296106 2775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:48:59.295965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36879 (local address 127.0.0.1:60750)
0926 20:48:59.296035 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0926 20:48:59.296066 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:36879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 20:48:59.296260 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 2476
I20260926 20:48:59.302256 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 2092
2026-09-26T20:48:59Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5584 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:48:59 dist-test-slave-hwlj krb5kdc[2778](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:48:59 dist-test-slave-hwlj krb5kdc[2778](info): set up 2 sockets
Sep 26 20:48:59 dist-test-slave-hwlj krb5kdc[2778](info): commencing operation
krb5kdc: starting...
W20260926 20:49:00.347679 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/krb5kdc/test-user.keytab.
Sep 26 20:49:00 dist-test-slave-hwlj krb5kdc[2778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:49:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:49:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 20:49:00.468897 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42383
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42383
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:49:00.551527 2791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:00.551740 2791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:00.551783 2791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:00.553010 2791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:49:00.553054 2791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:00.553069 2791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:00.553081 2791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:49:00.553144 2791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:49:00.554652 2791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:00.554937 2791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:00.555176 2791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:00.560024 2799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:00.560060 2791 server_base.cc:1034] running on GCE node
W20260926 20:49:00.560029 2796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:00.560122 2797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:00.560428 2791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:00.560756 2791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:00.561906 2791 hybrid_clock.cc:648] HybridClock initialized: now 1790455740561894 us; error 30 us; skew 500 ppm
Sep 26 20:49:00 dist-test-slave-hwlj krb5kdc[2778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455740, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:49:00.564416 2791 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:49:00.564824 2791 webserver.cc:460] Webserver started at http://127.0.98.62:42627/ using document root <none> and password file <none>
I20260926 20:49:00.565038 2791 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:00.565104 2791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:00.565254 2791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:49:00.566305 2791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/master-0/data/instance:
uuid: "f28eba1bfbc248369c105ff313d93a9e"
format_stamp: "Formatted at 2026-09-26 20:49:00 on dist-test-slave-hwlj"
server_key: "3c453d58c76782e85ffd6201cd13b82e"
server_key_iv: "4ec115791564f94125809f0e5cee1913"
server_key_version: "encryptionkey@0"
I20260926 20:49:00.566654 2791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/master-0/wal/instance:
uuid: "f28eba1bfbc248369c105ff313d93a9e"
format_stamp: "Formatted at 2026-09-26 20:49:00 on dist-test-slave-hwlj"
server_key: "3c453d58c76782e85ffd6201cd13b82e"
server_key_iv: "4ec115791564f94125809f0e5cee1913"
server_key_version: "encryptionkey@0"
I20260926 20:49:00.568048 2791 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 20:49:00.568791 2806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:00.568959 2791 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:49:00.569059 2791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "f28eba1bfbc248369c105ff313d93a9e"
format_stamp: "Formatted at 2026-09-26 20:49:00 on dist-test-slave-hwlj"
server_key: "3c453d58c76782e85ffd6201cd13b82e"
server_key_iv: "4ec115791564f94125809f0e5cee1913"
server_key_version: "encryptionkey@0"
I20260926 20:49:00.569131 2791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:00.582554 2791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:00.584699 2791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:00.584860 2791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:00.588874 2791 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42383
I20260926 20:49:00.588909 2858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42383 every 8 connection(s)
I20260926 20:49:00.589406 2791 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:49:00.590166 2859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:49:00.592497 2859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f28eba1bfbc248369c105ff313d93a9e: Bootstrap starting.
I20260926 20:49:00.593093 2859 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f28eba1bfbc248369c105ff313d93a9e: Neither blocks nor log segments found. Creating new log.
I20260926 20:49:00.593391 2859 log.cc:824] T 00000000000000000000000000000000 P f28eba1bfbc248369c105ff313d93a9e: Log is configured to *not* fsync() on all Append() calls
I20260926 20:49:00.593801 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 2791
I20260926 20:49:00.593902 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/master-0/wal/instance
I20260926 20:49:00.594103 392 external_mini_cluster.cc:1428] Setting key 166f1772ed4da8c275d7482be7399204
I20260926 20:49:00.594255 2859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f28eba1bfbc248369c105ff313d93a9e: No bootstrap required, opened a new log
I20260926 20:49:00.597121 2859 raft_consensus.cc:348] T 00000000000000000000000000000000 P f28eba1bfbc248369c105ff313d93a9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f28eba1bfbc248369c105ff313d93a9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42383 } }
I20260926 20:49:00.597285 2859 raft_consensus.cc:374] T 00000000000000000000000000000000 P f28eba1bfbc248369c105ff313d93a9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:49:00.597317 2859 raft_consensus.cc:729] T 00000000000000000000000000000000 P f28eba1bfbc248369c105ff313d93a9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f28eba1bfbc248369c105ff313d93a9e, State: Initialized, Role: FOLLOWER
I20260926 20:49:00.597434 2859 consensus_queue.cc:260] T 00000000000000000000000000000000 P f28eba1bfbc248369c105ff313d93a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f28eba1bfbc248369c105ff313d93a9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42383 } }
I20260926 20:49:00.597503 2859 raft_consensus.cc:388] T 00000000000000000000000000000000 P f28eba1bfbc248369c105ff313d93a9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:49:00.597530 2859 raft_consensus.cc:482] T 00000000000000000000000000000000 P f28eba1bfbc248369c105ff313d93a9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:49:00.597565 2859 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f28eba1bfbc248369c105ff313d93a9e [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:49:00.598206 2859 raft_consensus.cc:504] T 00000000000000000000000000000000 P f28eba1bfbc248369c105ff313d93a9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f28eba1bfbc248369c105ff313d93a9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42383 } }
I20260926 20:49:00.598294 2859 leader_election.cc:302] T 00000000000000000000000000000000 P f28eba1bfbc248369c105ff313d93a9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f28eba1bfbc248369c105ff313d93a9e; no voters:
I20260926 20:49:00.598466 2859 leader_election.cc:288] T 00000000000000000000000000000000 P f28eba1bfbc248369c105ff313d93a9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:49:00.598520 2864 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f28eba1bfbc248369c105ff313d93a9e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:49:00.598665 2864 raft_consensus.cc:686] T 00000000000000000000000000000000 P f28eba1bfbc248369c105ff313d93a9e [term 1 LEADER]: Becoming Leader. State: Replica: f28eba1bfbc248369c105ff313d93a9e, State: Running, Role: LEADER
I20260926 20:49:00.598749 2864 consensus_queue.cc:237] T 00000000000000000000000000000000 P f28eba1bfbc248369c105ff313d93a9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f28eba1bfbc248369c105ff313d93a9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42383 } }
I20260926 20:49:00.598752 2859 sys_catalog.cc:564] T 00000000000000000000000000000000 P f28eba1bfbc248369c105ff313d93a9e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:49:00.600450 2866 sys_catalog.cc:455] T 00000000000000000000000000000000 P f28eba1bfbc248369c105ff313d93a9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f28eba1bfbc248369c105ff313d93a9e. Latest consensus state: current_term: 1 leader_uuid: "f28eba1bfbc248369c105ff313d93a9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f28eba1bfbc248369c105ff313d93a9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42383 } } }
I20260926 20:49:00.600482 2865 sys_catalog.cc:455] T 00000000000000000000000000000000 P f28eba1bfbc248369c105ff313d93a9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f28eba1bfbc248369c105ff313d93a9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f28eba1bfbc248369c105ff313d93a9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42383 } } }
I20260926 20:49:00.600711 2866 sys_catalog.cc:458] T 00000000000000000000000000000000 P f28eba1bfbc248369c105ff313d93a9e [sys.catalog]: This master's current role is: LEADER
I20260926 20:49:00.600718 2865 sys_catalog.cc:458] T 00000000000000000000000000000000 P f28eba1bfbc248369c105ff313d93a9e [sys.catalog]: This master's current role is: LEADER
W20260926 20:49:00.601006 2879 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f28eba1bfbc248369c105ff313d93a9e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:49:00.601068 2879 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:49:00.601130 2881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:49:00.601303 2881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 20:49:00 dist-test-slave-hwlj krb5kdc[2778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:49:00.602977 2881 catalog_manager.cc:1347] Generated new cluster ID: aca2c8b49f1340d8a83495e065d0855c
I20260926 20:49:00.603030 2881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:49:00.605979 2863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:00.594598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54436 (local address 127.0.98.62:42383)
0926 20:49:00.594791 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:00.594801 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:00.594840 (+ 39us) server_negotiation.cc:407] Connection header received
0926 20:49:00.595658 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:00.595668 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:00.595734 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:00.595900 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:00.596485 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:00.597865 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:00.598961 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:00.599532 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:00.599641 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:00.602690 (+ 3049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:00.602694 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:00.602694 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:00.602705 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:00.604465 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:00.604839 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:00.604842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:00.604842 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:00.604933 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:00.605154 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:00.605155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:00.605155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:00.605388 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:00.605555 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:00.605662 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 20:49:00.605796 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":87,"threads_started":1}
I20260926 20:49:00.652287 2881 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:49:00.652875 2881 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:49:00.657691 2881 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f28eba1bfbc248369c105ff313d93a9e: Generated new TSK 0
I20260926 20:49:00.657879 2881 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 20:49:00.691670 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42383
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:49:00.772295 2885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:00.772490 2885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:00.772513 2885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:00.773691 2885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:00.773731 2885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:00.773757 2885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:49:00.775185 2885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:00.775369 2885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:00.775537 2885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:00.780643 2893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:00.780651 2890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:00.780651 2891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:00.780743 2885 server_base.cc:1034] running on GCE node
I20260926 20:49:00.781049 2885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:00.781296 2885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:00.782449 2885 hybrid_clock.cc:648] HybridClock initialized: now 1790455740782442 us; error 39 us; skew 500 ppm
Sep 26 20:49:00 dist-test-slave-hwlj krb5kdc[2778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455740, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:49:00.785209 2885 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 20:49:00.785597 2885 webserver.cc:460] Webserver started at http://127.0.98.1:32805/ using document root <none> and password file <none>
I20260926 20:49:00.785797 2885 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:00.785838 2885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:00.785926 2885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:49:00.787005 2885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-0/data/instance:
uuid: "ff5363c2e8074f478c7d30c187c813a0"
format_stamp: "Formatted at 2026-09-26 20:49:00 on dist-test-slave-hwlj"
server_key: "1cc98049f75947e6a87d9475385ddd05"
server_key_iv: "2ec95077531c6c3d1bcaae0c62004af5"
server_key_version: "encryptionkey@0"
I20260926 20:49:00.787314 2885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ff5363c2e8074f478c7d30c187c813a0"
format_stamp: "Formatted at 2026-09-26 20:49:00 on dist-test-slave-hwlj"
server_key: "1cc98049f75947e6a87d9475385ddd05"
server_key_iv: "2ec95077531c6c3d1bcaae0c62004af5"
server_key_version: "encryptionkey@0"
I20260926 20:49:00.788653 2885 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:49:00.789402 2900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:00.789590 2885 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:49:00.789662 2885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-0/wal
uuid: "ff5363c2e8074f478c7d30c187c813a0"
format_stamp: "Formatted at 2026-09-26 20:49:00 on dist-test-slave-hwlj"
server_key: "1cc98049f75947e6a87d9475385ddd05"
server_key_iv: "2ec95077531c6c3d1bcaae0c62004af5"
server_key_version: "encryptionkey@0"
I20260926 20:49:00.789752 2885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:00.796973 2885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:00.799254 2885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:00.799394 2885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:00.799647 2885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:49:00.799986 2885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:49:00.800033 2885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:00.800057 2885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:49:00.800071 2885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:00.805644 2885 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36561
I20260926 20:49:00.805722 3013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36561 every 8 connection(s)
I20260926 20:49:00.806036 2885 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:49:00.806483 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 2885
I20260926 20:49:00.806588 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-0/wal/instance
I20260926 20:49:00.806777 392 external_mini_cluster.cc:1428] Setting key 36e3aa63dd736dcc8257be5f1277f72f
Sep 26 20:49:00 dist-test-slave-hwlj krb5kdc[2778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455740, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 20:49:00.817014 2863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:00.806786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58669 (local address 127.0.98.62:42383)
0926 20:49:00.806860 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:00.806862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:00.807068 (+ 206us) server_negotiation.cc:407] Connection header received
0926 20:49:00.808395 (+ 1327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:00.808396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:00.808406 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:00.808458 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:00.809302 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:00.810059 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:00.811016 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:00.811392 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:00.811476 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:00.814075 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:00.814076 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:00.814077 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:00.814082 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:00.815491 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:00.816022 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:00.816023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:00.816024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:00.816074 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:00.816351 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:00.816351 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:00.816351 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:00.816453 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:00.816501 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:00.816812 (+ 311us) server_negotiation.cc:299] Negotiation successful
0926 20:49:00.816926 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 20:49:00.817265 3016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:00.806784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.1:58669)
0926 20:49:00.806948 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:00.806987 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:00.808291 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:00.808530 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:00.808534 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:00.808924 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:00.809189 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:00.809194 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:00.810143 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:00.810143 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:00.810923 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:00.810925 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:00.811517 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:00.811518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:00.811632 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:00.812265 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:00.812274 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:00.813970 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:00.815610 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:00.815618 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:00.815621 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:00.815955 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:00.816164 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:00.816165 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:00.816165 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:00.816277 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:00.816509 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:00.816513 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:00.816649 (+ 136us) client_negotiation.cc:769] Sending connection context
0926 20:49:00.816757 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 20:49:00.816949 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":100,"threads_started":1}
I20260926 20:49:00.817816 3014 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42383
I20260926 20:49:00.817924 3014 heartbeater.cc:464] Registering TS with master...
I20260926 20:49:00.818065 3014 heartbeater.cc:511] Master 127.0.98.62:42383 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 20:49:00.818640 2823 ts_manager.cc:194] Registered new tserver with Master: ff5363c2e8074f478c7d30c187c813a0 (127.0.98.1:36561)
I20260926 20:49:00.819321 2823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58669
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 20:49:00.835222 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42383
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:49:00.915050 3019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:00.915241 3019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:00.915263 3019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:00.916426 3019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:00.916465 3019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:00.916491 3019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:49:00.917943 3019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:00.918186 3019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:00.918406 3019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:00.923339 3027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:00.923537 3024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:00.923470 3019 server_base.cc:1034] running on GCE node
W20260926 20:49:00.923410 3025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:00.923913 3019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:00.924253 3019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:00.925444 3019 hybrid_clock.cc:648] HybridClock initialized: now 1790455740925440 us; error 37 us; skew 500 ppm
Sep 26 20:49:00 dist-test-slave-hwlj krb5kdc[2778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455740, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:49:00.928071 3019 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 20:49:00.928557 3019 webserver.cc:460] Webserver started at http://127.0.98.2:43155/ using document root <none> and password file <none>
I20260926 20:49:00.928820 3019 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:00.928891 3019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:00.929046 3019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:49:00.930174 3019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-1/data/instance:
uuid: "84fa6e061119458b9431acc5cc5e88ca"
format_stamp: "Formatted at 2026-09-26 20:49:00 on dist-test-slave-hwlj"
server_key: "57b4cdbec1b646d837519d752b11f0cf"
server_key_iv: "69cfd27c06a34e60b571857d996c757c"
server_key_version: "encryptionkey@0"
I20260926 20:49:00.930557 3019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-1/wal/instance:
uuid: "84fa6e061119458b9431acc5cc5e88ca"
format_stamp: "Formatted at 2026-09-26 20:49:00 on dist-test-slave-hwlj"
server_key: "57b4cdbec1b646d837519d752b11f0cf"
server_key_iv: "69cfd27c06a34e60b571857d996c757c"
server_key_version: "encryptionkey@0"
I20260926 20:49:00.931984 3019 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:49:00.932821 3034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:00.933034 3019 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:49:00.933140 3019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-1/wal
uuid: "84fa6e061119458b9431acc5cc5e88ca"
format_stamp: "Formatted at 2026-09-26 20:49:00 on dist-test-slave-hwlj"
server_key: "57b4cdbec1b646d837519d752b11f0cf"
server_key_iv: "69cfd27c06a34e60b571857d996c757c"
server_key_version: "encryptionkey@0"
I20260926 20:49:00.933225 3019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:00.939805 3019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:00.941748 3019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:00.941905 3019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:00.942130 3019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:49:00.942418 3019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:49:00.942471 3019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:00.942512 3019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:49:00.942541 3019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:00.948294 3019 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37487
I20260926 20:49:00.948335 3147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37487 every 8 connection(s)
I20260926 20:49:00.948644 3019 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
I20260926 20:49:00.949735 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 3019
I20260926 20:49:00.949808 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-1/wal/instance
I20260926 20:49:00.950012 392 external_mini_cluster.cc:1428] Setting key 7d9ee794eb9c6cf21d7bb75f013bdae5
Sep 26 20:49:00 dist-test-slave-hwlj krb5kdc[2778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455740, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 20:49:00.959260 2863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:00.949368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41413 (local address 127.0.98.62:42383)
0926 20:49:00.949455 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:00.949457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:00.949688 (+ 231us) server_negotiation.cc:407] Connection header received
0926 20:49:00.950930 (+ 1242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:00.950932 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:00.950943 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:00.950982 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:00.951635 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:00.952481 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:00.953519 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:00.953840 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:00.953922 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:00.956203 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:00.956204 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:00.956204 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:00.956212 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:00.957896 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:00.958396 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:00.958396 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:00.958397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:00.958447 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:00.958671 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:00.958672 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:00.958672 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:00.958768 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:00.958819 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:00.959080 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 20:49:00.959161 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 20:49:00.959472 3150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:00.949311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.2:41413)
0926 20:49:00.949532 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:00.949594 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:00.950827 (+ 1233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:00.951061 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:00.951065 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:00.951332 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:00.951524 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:00.951528 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:00.952584 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:00.952585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:00.953418 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:00.953420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:00.953949 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:00.953950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:00.954102 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:00.954690 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:00.954698 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:00.956098 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:00.958065 (+ 1967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:00.958072 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:00.958074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:00.958318 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:00.958517 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:00.958518 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:00.958518 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:00.958612 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:00.958823 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:00.958827 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:00.958938 (+ 111us) client_negotiation.cc:769] Sending connection context
0926 20:49:00.959055 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 20:49:00.959242 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":96,"threads_started":1}
I20260926 20:49:00.960139 3148 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42383
I20260926 20:49:00.960234 3148 heartbeater.cc:464] Registering TS with master...
I20260926 20:49:00.960368 3148 heartbeater.cc:511] Master 127.0.98.62:42383 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 20:49:00.960844 2823 ts_manager.cc:194] Registered new tserver with Master: 84fa6e061119458b9431acc5cc5e88ca (127.0.98.2:37487)
I20260926 20:49:00.961231 2823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41413
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 20:49:00.976136 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42383
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:49:01.056188 3153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:01.056380 3153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:01.056402 3153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:01.057582 3153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:01.057623 3153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:01.057649 3153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:49:01.059115 3153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:01.059352 3153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:01.059592 3153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:01.064935 3161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:01.064908 3159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:01.064906 3158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:01.065141 3153 server_base.cc:1034] running on GCE node
I20260926 20:49:01.065346 3153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:01.065541 3153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:01.066711 3153 hybrid_clock.cc:648] HybridClock initialized: now 1790455741066697 us; error 27 us; skew 500 ppm
Sep 26 20:49:01 dist-test-slave-hwlj krb5kdc[2778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455741, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:49:01.069535 3153 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 20:49:01.069909 3153 webserver.cc:460] Webserver started at http://127.0.98.3:35785/ using document root <none> and password file <none>
I20260926 20:49:01.070091 3153 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:01.070130 3153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:01.070217 3153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:49:01.071278 3153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-2/data/instance:
uuid: "fc3eb2dc5d8c46b1b9d8239f1aef0982"
format_stamp: "Formatted at 2026-09-26 20:49:01 on dist-test-slave-hwlj"
server_key: "8ff2b610bbef7c9fd275671ced224711"
server_key_iv: "c72b54fdd078e21d83cc09eb4c9fd255"
server_key_version: "encryptionkey@0"
I20260926 20:49:01.071590 3153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fc3eb2dc5d8c46b1b9d8239f1aef0982"
format_stamp: "Formatted at 2026-09-26 20:49:01 on dist-test-slave-hwlj"
server_key: "8ff2b610bbef7c9fd275671ced224711"
server_key_iv: "c72b54fdd078e21d83cc09eb4c9fd255"
server_key_version: "encryptionkey@0"
I20260926 20:49:01.073099 3153 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:49:01.073865 3168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:01.074100 3153 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:49:01.074208 3153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-2/wal
uuid: "fc3eb2dc5d8c46b1b9d8239f1aef0982"
format_stamp: "Formatted at 2026-09-26 20:49:01 on dist-test-slave-hwlj"
server_key: "8ff2b610bbef7c9fd275671ced224711"
server_key_iv: "c72b54fdd078e21d83cc09eb4c9fd255"
server_key_version: "encryptionkey@0"
I20260926 20:49:01.074287 3153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:01.084061 3153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:01.086048 3153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:01.086201 3153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:01.086423 3153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:49:01.086715 3153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:49:01.086746 3153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:01.086767 3153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:49:01.086800 3153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:01.092329 3153 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34299
I20260926 20:49:01.092404 3281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34299 every 8 connection(s)
I20260926 20:49:01.092706 3153 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 20:49:01 dist-test-slave-hwlj krb5kdc[2778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455741, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:49:01.100879 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 3153
I20260926 20:49:01.100986 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-2/wal/instance
I20260926 20:49:01.101181 392 external_mini_cluster.cc:1428] Setting key a5d89c3a91c556b5f85f4d36c7086d3b
I20260926 20:49:01.102259 2863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:01.093537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50769 (local address 127.0.98.62:42383)
0926 20:49:01.093624 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:01.093627 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:01.093804 (+ 177us) server_negotiation.cc:407] Connection header received
0926 20:49:01.094704 (+ 900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:01.094705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:01.094715 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:01.094770 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:01.095326 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:01.095901 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:01.096862 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:01.097055 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:01.097112 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:01.099243 (+ 2131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:01.099245 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:01.099245 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:01.099253 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:01.100751 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:01.101294 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:01.101295 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:01.101296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:01.101360 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:01.101621 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:01.101622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:01.101623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:01.101743 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:01.101820 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:01.102081 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 20:49:01.102159 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 20:49:01.102442 3284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:01.093497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42383 (local address 127.0.98.3:50769)
0926 20:49:01.093675 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:01.093711 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:01.094607 (+ 896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:01.094823 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:01.094826 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:01.095065 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:01.095252 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:01.095256 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:01.095979 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:01.095980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:01.096783 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:01.096785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:01.097131 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:01.097131 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:01.097243 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:01.097673 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:01.097682 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:01.099110 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:01.100834 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:01.100841 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:01.100844 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:01.101198 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:01.101437 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:01.101438 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:01.101439 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:01.101545 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:01.101812 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:01.101818 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:01.101954 (+ 136us) client_negotiation.cc:769] Sending connection context
0926 20:49:01.102060 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 20:49:01.102217 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":119,"threads_started":1}
I20260926 20:49:01.102913 3282 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42383
I20260926 20:49:01.103005 3282 heartbeater.cc:464] Registering TS with master...
I20260926 20:49:01.103149 3282 heartbeater.cc:511] Master 127.0.98.62:42383 requested a full tablet report, sending...
I20260926 20:49:01.103483 2823 ts_manager.cc:194] Registered new tserver with Master: fc3eb2dc5d8c46b1b9d8239f1aef0982 (127.0.98.3:34299)
I20260926 20:49:01.103886 2823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50769
I20260926 20:49:01.112967 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 20:49:01.123749 2863 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:01.122736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54440 (local address 127.0.98.62:42383)
0926 20:49:01.122805 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:01.122807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:01.122915 (+ 108us) server_negotiation.cc:407] Connection header received
0926 20:49:01.122957 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:01.122958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:01.122967 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:01.123080 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:01.123656 (+ 576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54440: BlockingRecv error: recv got EOF from 127.0.0.1:54440 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 20:49:01.124650 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 2885
I20260926 20:49:01.129304 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 3019
I20260926 20:49:01.133460 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 3153
I20260926 20:49:01.137614 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 2791
2026-09-26T20:49:01Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1832 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455725523895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:49:01 dist-test-slave-hwlj krb5kdc[3294](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:49:01 dist-test-slave-hwlj krb5kdc[3294](info): set up 2 sockets
Sep 26 20:49:01 dist-test-slave-hwlj krb5kdc[3294](info): commencing operation
krb5kdc: starting...
W20260926 20:49:02.187712 392 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455725523895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455725523895-392-0/krb5kdc/test-user.keytab.
Sep 26 20:49:02 dist-test-slave-hwlj krb5kdc[3294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:49:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:49:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 20:49:02.314483 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43471
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:49:02.396061 3307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:02.396265 3307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:02.396288 3307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:02.397388 3307 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:49:02.397423 3307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:02.397436 3307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:02.397449 3307 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:49:02.397464 3307 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:49:02.398872 3307 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455725523895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:02.399057 3307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:02.399219 3307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:49:02.403928 3307 server_base.cc:1034] running on GCE node
W20260926 20:49:02.403913 3312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:02.403896 3313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:02.403913 3315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:02.404361 3307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:02.404713 3307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:02.405915 3307 hybrid_clock.cc:648] HybridClock initialized: now 1790455742405903 us; error 30 us; skew 500 ppm
Sep 26 20:49:02 dist-test-slave-hwlj krb5kdc[3294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455742, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:49:02.408521 3307 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:49:02.409006 3307 webserver.cc:460] Webserver started at http://127.0.98.62:43957/ using document root <none> and password file <none>
I20260926 20:49:02.409241 3307 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:02.409312 3307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:02.409475 3307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:49:02.410514 3307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455725523895-392-0/minicluster-data/master-0/data/instance:
uuid: "f1e144326ea24a0fb1db2672e38057ad"
format_stamp: "Formatted at 2026-09-26 20:49:02 on dist-test-slave-hwlj"
server_key: "032879fb724282806f64ca05f8b4b11b"
server_key_iv: "6d2c2866467ef7887a77db9e4a630b55"
server_key_version: "encryptionkey@0"
I20260926 20:49:02.410887 3307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455725523895-392-0/minicluster-data/master-0/wal/instance:
uuid: "f1e144326ea24a0fb1db2672e38057ad"
format_stamp: "Formatted at 2026-09-26 20:49:02 on dist-test-slave-hwlj"
server_key: "032879fb724282806f64ca05f8b4b11b"
server_key_iv: "6d2c2866467ef7887a77db9e4a630b55"
server_key_version: "encryptionkey@0"
I20260926 20:49:02.412330 3307 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:49:02.413141 3322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:02.413355 3307 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:49:02.413471 3307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "f1e144326ea24a0fb1db2672e38057ad"
format_stamp: "Formatted at 2026-09-26 20:49:02 on dist-test-slave-hwlj"
server_key: "032879fb724282806f64ca05f8b4b11b"
server_key_iv: "6d2c2866467ef7887a77db9e4a630b55"
server_key_version: "encryptionkey@0"
I20260926 20:49:02.413549 3307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:02.433492 3307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:02.436007 3307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:02.436182 3307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:02.440255 3307 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43471
I20260926 20:49:02.440310 3374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43471 every 8 connection(s)
I20260926 20:49:02.440685 3307 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:49:02.441535 3375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:49:02.443667 3375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1e144326ea24a0fb1db2672e38057ad: Bootstrap starting.
I20260926 20:49:02.444283 3375 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1e144326ea24a0fb1db2672e38057ad: Neither blocks nor log segments found. Creating new log.
I20260926 20:49:02.444514 3375 log.cc:824] T 00000000000000000000000000000000 P f1e144326ea24a0fb1db2672e38057ad: Log is configured to *not* fsync() on all Append() calls
I20260926 20:49:02.445106 3375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1e144326ea24a0fb1db2672e38057ad: No bootstrap required, opened a new log
I20260926 20:49:02.446604 3375 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1e144326ea24a0fb1db2672e38057ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1e144326ea24a0fb1db2672e38057ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43471 } }
I20260926 20:49:02.446739 3375 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1e144326ea24a0fb1db2672e38057ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:49:02.446762 3375 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1e144326ea24a0fb1db2672e38057ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e144326ea24a0fb1db2672e38057ad, State: Initialized, Role: FOLLOWER
I20260926 20:49:02.446905 3375 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1e144326ea24a0fb1db2672e38057ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1e144326ea24a0fb1db2672e38057ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43471 } }
I20260926 20:49:02.446992 3375 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1e144326ea24a0fb1db2672e38057ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:49:02.447016 3375 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1e144326ea24a0fb1db2672e38057ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:49:02.447043 3375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1e144326ea24a0fb1db2672e38057ad [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:49:02.447849 3375 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1e144326ea24a0fb1db2672e38057ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1e144326ea24a0fb1db2672e38057ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43471 } }
I20260926 20:49:02.447933 3375 leader_election.cc:302] T 00000000000000000000000000000000 P f1e144326ea24a0fb1db2672e38057ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1e144326ea24a0fb1db2672e38057ad; no voters:
I20260926 20:49:02.448052 3375 leader_election.cc:288] T 00000000000000000000000000000000 P f1e144326ea24a0fb1db2672e38057ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:49:02.448155 3378 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1e144326ea24a0fb1db2672e38057ad [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:49:02.448307 3378 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1e144326ea24a0fb1db2672e38057ad [term 1 LEADER]: Becoming Leader. State: Replica: f1e144326ea24a0fb1db2672e38057ad, State: Running, Role: LEADER
I20260926 20:49:02.448314 3375 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1e144326ea24a0fb1db2672e38057ad [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:49:02.448395 3378 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1e144326ea24a0fb1db2672e38057ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1e144326ea24a0fb1db2672e38057ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43471 } }
I20260926 20:49:02.448727 3380 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1e144326ea24a0fb1db2672e38057ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1e144326ea24a0fb1db2672e38057ad. Latest consensus state: current_term: 1 leader_uuid: "f1e144326ea24a0fb1db2672e38057ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1e144326ea24a0fb1db2672e38057ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43471 } } }
I20260926 20:49:02.448825 3380 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1e144326ea24a0fb1db2672e38057ad [sys.catalog]: This master's current role is: LEADER
I20260926 20:49:02.448726 3379 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1e144326ea24a0fb1db2672e38057ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1e144326ea24a0fb1db2672e38057ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1e144326ea24a0fb1db2672e38057ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43471 } } }
I20260926 20:49:02.448971 3379 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1e144326ea24a0fb1db2672e38057ad [sys.catalog]: This master's current role is: LEADER
I20260926 20:49:02.449044 3385 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:49:02.449482 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 3307
I20260926 20:49:02.449532 3385 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:49:02.449576 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455725523895-392-0/minicluster-data/master-0/wal/instance
I20260926 20:49:02.449755 392 external_mini_cluster.cc:1428] Setting key 290253d15868a8aa454ee02fd29e9b31
I20260926 20:49:02.451436 3385 catalog_manager.cc:1347] Generated new cluster ID: ef18000bbc3147f29b7e85524a2ae022
I20260926 20:49:02.451489 3385 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 20:49:02 dist-test-slave-hwlj krb5kdc[3294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:49:02.461788 3397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:02.450367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47590 (local address 127.0.98.62:43471)
0926 20:49:02.450654 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:02.450679 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:02.450854 (+ 175us) server_negotiation.cc:407] Connection header received
0926 20:49:02.451086 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:02.451095 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:02.451165 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:02.451314 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:02.452090 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:02.453439 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:02.454520 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:02.455151 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:02.455244 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:02.458113 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:02.458117 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:02.458117 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:02.458129 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:02.460132 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:02.460529 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:02.460532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:02.460532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:02.460649 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:02.460851 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:02.460852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:02.460852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:02.461177 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:02.461333 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:02.461429 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 20:49:02.461563 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":61,"threads_started":1}
I20260926 20:49:02.462862 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:49:02.465538 3385 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:49:02.466030 3385 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:49:02.469494 3385 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1e144326ea24a0fb1db2672e38057ad: Generated new TSK 0
I20260926 20:49:02.469664 3385 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:49:02.481773 3397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:02.477715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47596 (local address 127.0.98.62:43471)
0926 20:49:02.477779 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:02.477781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:02.477874 (+ 93us) server_negotiation.cc:407] Connection header received
0926 20:49:02.477908 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:02.477909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:02.477921 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:02.477993 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:02.478332 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:02.479118 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:02.480098 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:02.480434 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:02.480507 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:02.481271 (+ 764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:02.481272 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:02.481272 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:49:02.481492 (+ 220us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:02.481510 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:49:02.481517 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:02.481579 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:02.481672 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 20:49:02.481697 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 20:49:02.484182 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 3307
2026-09-26T20:49:02Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1344 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:49:02 dist-test-slave-hwlj krb5kdc[3408](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:49:02 dist-test-slave-hwlj krb5kdc[3408](info): set up 2 sockets
Sep 26 20:49:02 dist-test-slave-hwlj krb5kdc[3408](info): commencing operation
krb5kdc: starting...
W20260926 20:49:03.519582 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/krb5kdc/test-user.keytab.
Sep 26 20:49:03 dist-test-slave-hwlj krb5kdc[3408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:49:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:49:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 20:49:03.639252 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43515
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43515
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:49:03.719870 3421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:03.720089 3421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:03.720120 3421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:03.721225 3421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:49:03.721257 3421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:03.721272 3421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:03.721284 3421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:49:03.721297 3421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:49:03.722689 3421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:03.722885 3421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:03.723058 3421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:03.728034 3426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:03.728034 3427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:03.728063 3421 server_base.cc:1034] running on GCE node
W20260926 20:49:03.728058 3429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:03.728591 3421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:03.728911 3421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:03.730106 3421 hybrid_clock.cc:648] HybridClock initialized: now 1790455743730089 us; error 28 us; skew 500 ppm
Sep 26 20:49:03 dist-test-slave-hwlj krb5kdc[3408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:49:03.739944 3421 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:49:03.740517 3421 webserver.cc:460] Webserver started at http://127.0.98.62:44207/ using document root <none> and password file <none>
I20260926 20:49:03.740718 3421 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:03.740760 3421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:03.740859 3421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:49:03.742219 3421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/master-0/data/instance:
uuid: "4aa56ff1153c4119a6693ce579cb4f21"
format_stamp: "Formatted at 2026-09-26 20:49:03 on dist-test-slave-hwlj"
server_key: "a54e39d58abd6d1f811938a026ab4476"
server_key_iv: "bde42a8150c73e0997e3f57eace6bb95"
server_key_version: "encryptionkey@0"
I20260926 20:49:03.742558 3421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/master-0/wal/instance:
uuid: "4aa56ff1153c4119a6693ce579cb4f21"
format_stamp: "Formatted at 2026-09-26 20:49:03 on dist-test-slave-hwlj"
server_key: "a54e39d58abd6d1f811938a026ab4476"
server_key_iv: "bde42a8150c73e0997e3f57eace6bb95"
server_key_version: "encryptionkey@0"
I20260926 20:49:03.744102 3421 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:49:03.745059 3436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:03.745285 3421 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:49:03.745391 3421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "4aa56ff1153c4119a6693ce579cb4f21"
format_stamp: "Formatted at 2026-09-26 20:49:03 on dist-test-slave-hwlj"
server_key: "a54e39d58abd6d1f811938a026ab4476"
server_key_iv: "bde42a8150c73e0997e3f57eace6bb95"
server_key_version: "encryptionkey@0"
I20260926 20:49:03.745456 3421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:03.754817 3421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:03.756924 3421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:03.757071 3421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:03.760731 3421 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43515
I20260926 20:49:03.760787 3488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43515 every 8 connection(s)
I20260926 20:49:03.761114 3421 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:49:03.762107 3489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:49:03.763953 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 3421
I20260926 20:49:03.764045 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/master-0/wal/instance
I20260926 20:49:03.764233 392 external_mini_cluster.cc:1428] Setting key 8f6413ffa0974735ab33128a0c816e5c
I20260926 20:49:03.764688 3489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4aa56ff1153c4119a6693ce579cb4f21: Bootstrap starting.
I20260926 20:49:03.765413 3489 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4aa56ff1153c4119a6693ce579cb4f21: Neither blocks nor log segments found. Creating new log.
I20260926 20:49:03.765725 3489 log.cc:824] T 00000000000000000000000000000000 P 4aa56ff1153c4119a6693ce579cb4f21: Log is configured to *not* fsync() on all Append() calls
I20260926 20:49:03.766395 3489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4aa56ff1153c4119a6693ce579cb4f21: No bootstrap required, opened a new log
I20260926 20:49:03.768702 3489 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4aa56ff1153c4119a6693ce579cb4f21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aa56ff1153c4119a6693ce579cb4f21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43515 } }
I20260926 20:49:03.768867 3489 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4aa56ff1153c4119a6693ce579cb4f21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:49:03.768899 3489 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4aa56ff1153c4119a6693ce579cb4f21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aa56ff1153c4119a6693ce579cb4f21, State: Initialized, Role: FOLLOWER
I20260926 20:49:03.769057 3489 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4aa56ff1153c4119a6693ce579cb4f21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aa56ff1153c4119a6693ce579cb4f21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43515 } }
I20260926 20:49:03.769141 3489 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4aa56ff1153c4119a6693ce579cb4f21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:49:03.769171 3489 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4aa56ff1153c4119a6693ce579cb4f21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:49:03.769212 3489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4aa56ff1153c4119a6693ce579cb4f21 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:49:03.769924 3489 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4aa56ff1153c4119a6693ce579cb4f21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aa56ff1153c4119a6693ce579cb4f21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43515 } }
I20260926 20:49:03.770021 3489 leader_election.cc:302] T 00000000000000000000000000000000 P 4aa56ff1153c4119a6693ce579cb4f21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4aa56ff1153c4119a6693ce579cb4f21; no voters:
I20260926 20:49:03.770212 3489 leader_election.cc:288] T 00000000000000000000000000000000 P 4aa56ff1153c4119a6693ce579cb4f21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:49:03.770231 3494 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4aa56ff1153c4119a6693ce579cb4f21 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:49:03.770345 3494 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4aa56ff1153c4119a6693ce579cb4f21 [term 1 LEADER]: Becoming Leader. State: Replica: 4aa56ff1153c4119a6693ce579cb4f21, State: Running, Role: LEADER
I20260926 20:49:03.770471 3489 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4aa56ff1153c4119a6693ce579cb4f21 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:49:03.770479 3494 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4aa56ff1153c4119a6693ce579cb4f21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aa56ff1153c4119a6693ce579cb4f21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43515 } }
I20260926 20:49:03.770876 3495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aa56ff1153c4119a6693ce579cb4f21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4aa56ff1153c4119a6693ce579cb4f21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aa56ff1153c4119a6693ce579cb4f21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43515 } } }
I20260926 20:49:03.770948 3495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aa56ff1153c4119a6693ce579cb4f21 [sys.catalog]: This master's current role is: LEADER
I20260926 20:49:03.771104 3496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aa56ff1153c4119a6693ce579cb4f21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4aa56ff1153c4119a6693ce579cb4f21. Latest consensus state: current_term: 1 leader_uuid: "4aa56ff1153c4119a6693ce579cb4f21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aa56ff1153c4119a6693ce579cb4f21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43515 } } }
I20260926 20:49:03.771166 3496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aa56ff1153c4119a6693ce579cb4f21 [sys.catalog]: This master's current role is: LEADER
Sep 26 20:49:03 dist-test-slave-hwlj krb5kdc[3408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:49:03.771919 3499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:49:03.772460 3499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:49:03.775816 3492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:03.764751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60494 (local address 127.0.98.62:43515)
0926 20:49:03.764911 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:03.764920 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:03.764951 (+ 31us) server_negotiation.cc:407] Connection header received
0926 20:49:03.765577 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:03.765588 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:03.765653 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:03.765806 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:03.766475 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:03.767887 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:03.768636 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:03.769225 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:03.769338 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:03.772393 (+ 3055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:03.772397 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:03.772397 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:03.772409 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:03.774199 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:03.774611 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:03.774613 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:03.774614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:03.774693 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:03.774907 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:03.774907 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:03.774908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:03.775156 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:03.775251 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:03.775404 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 20:49:03.775502 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":79,"threads_started":1}
I20260926 20:49:03.780004 3499 catalog_manager.cc:1347] Generated new cluster ID: 31a1842fc4a44b8e8a0307d18c22f1c8
I20260926 20:49:03.780056 3499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:49:03.787736 3499 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:49:03.788479 3499 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:49:03.791796 3499 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4aa56ff1153c4119a6693ce579cb4f21: Generated new TSK 0
I20260926 20:49:03.791966 3499 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 20:49:03.827850 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43515
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:49:03.923520 3515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:03.923740 3515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:03.923862 3515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:03.925241 3515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:03.925292 3515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:03.925320 3515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:49:03.926837 3515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:03.927099 3515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:03.927372 3515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:03.932518 3523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:03.932600 3521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:03.932518 3520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:03.932741 3515 server_base.cc:1034] running on GCE node
I20260926 20:49:03.933000 3515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:03.933310 3515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:03.934501 3515 hybrid_clock.cc:648] HybridClock initialized: now 1790455743934492 us; error 49 us; skew 500 ppm
Sep 26 20:49:03 dist-test-slave-hwlj krb5kdc[3408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:49:03.937433 3515 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 20:49:03.937927 3515 webserver.cc:460] Webserver started at http://127.0.98.1:42941/ using document root <none> and password file <none>
I20260926 20:49:03.938180 3515 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:03.938256 3515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:03.938410 3515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:49:03.939512 3515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-0/data/instance:
uuid: "cc64e98cb18b41ec995f6b3fe6e4134c"
format_stamp: "Formatted at 2026-09-26 20:49:03 on dist-test-slave-hwlj"
server_key: "02e5e03c376f2b60eafc11b034cc5b7f"
server_key_iv: "59ab51602050a5d29cceb31682ae5f90"
server_key_version: "encryptionkey@0"
I20260926 20:49:03.939931 3515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cc64e98cb18b41ec995f6b3fe6e4134c"
format_stamp: "Formatted at 2026-09-26 20:49:03 on dist-test-slave-hwlj"
server_key: "02e5e03c376f2b60eafc11b034cc5b7f"
server_key_iv: "59ab51602050a5d29cceb31682ae5f90"
server_key_version: "encryptionkey@0"
I20260926 20:49:03.941411 3515 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:49:03.942349 3530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:03.942613 3515 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:49:03.942727 3515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-0/wal
uuid: "cc64e98cb18b41ec995f6b3fe6e4134c"
format_stamp: "Formatted at 2026-09-26 20:49:03 on dist-test-slave-hwlj"
server_key: "02e5e03c376f2b60eafc11b034cc5b7f"
server_key_iv: "59ab51602050a5d29cceb31682ae5f90"
server_key_version: "encryptionkey@0"
I20260926 20:49:03.942821 3515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:03.958029 3515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:03.960573 3515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:03.960755 3515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:03.961050 3515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:49:03.961391 3515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:49:03.961450 3515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:03.961490 3515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:49:03.961540 3515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:03.967274 3515 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42265
I20260926 20:49:03.967393 3643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42265 every 8 connection(s)
I20260926 20:49:03.967689 3515 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:49:03.973629 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 3515
I20260926 20:49:03.973749 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-0/wal/instance
I20260926 20:49:03.973963 392 external_mini_cluster.cc:1428] Setting key 28cfca161d45014ac0d63b9a1ee67155
Sep 26 20:49:03 dist-test-slave-hwlj krb5kdc[3408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:49:03.979362 3492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:03.968649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54695 (local address 127.0.98.62:43515)
0926 20:49:03.968712 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:03.968714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:03.968903 (+ 189us) server_negotiation.cc:407] Connection header received
0926 20:49:03.969504 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:03.969506 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:03.969519 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:03.969593 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:03.970742 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:03.971496 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:03.972729 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:03.972982 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:03.973059 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:03.976103 (+ 3044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:03.976105 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:03.976106 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:03.976115 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:03.977727 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:03.978237 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:03.978238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:03.978239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:03.978306 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:03.978607 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:03.978608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:03.978609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:03.978739 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:03.978828 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:03.979144 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 20:49:03.979244 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 20:49:03.979643 3646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:03.968574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43515 (local address 127.0.98.1:54695)
0926 20:49:03.968763 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:03.968809 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:03.969371 (+ 562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:03.969714 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:03.969719 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:03.970377 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:03.970627 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:03.970633 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:03.971595 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:03.971596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:03.972634 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:03.972637 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:03.973108 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:03.973109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:03.973225 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:03.973908 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:03.973917 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:03.975993 (+ 2076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:03.977847 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:03.977856 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:03.977858 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:03.978131 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:03.978386 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:03.978387 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:03.978388 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:03.978527 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:03.978818 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:03.978825 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:03.978986 (+ 161us) client_negotiation.cc:769] Sending connection context
0926 20:49:03.979107 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 20:49:03.979335 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":104,"threads_started":1}
I20260926 20:49:03.980288 3644 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43515
I20260926 20:49:03.980392 3644 heartbeater.cc:464] Registering TS with master...
I20260926 20:49:03.980625 3644 heartbeater.cc:511] Master 127.0.98.62:43515 requested a full tablet report, sending...
I20260926 20:49:03.981195 3453 ts_manager.cc:194] Registered new tserver with Master: cc64e98cb18b41ec995f6b3fe6e4134c (127.0.98.1:42265)
I20260926 20:49:03.981890 3453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54695
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 20:49:04.003509 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43515
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:49:04.088048 3649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:04.088312 3649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:04.088338 3649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:04.089443 3649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:04.089479 3649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:04.089504 3649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:49:04.090834 3649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:04.091006 3649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:04.091171 3649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:04.096238 3654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:04.096238 3657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:04.096243 3655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:04.096316 3649 server_base.cc:1034] running on GCE node
I20260926 20:49:04.096761 3649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:04.097033 3649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:04.098287 3649 hybrid_clock.cc:648] HybridClock initialized: now 1790455744098262 us; error 43 us; skew 500 ppm
Sep 26 20:49:04 dist-test-slave-hwlj krb5kdc[3408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455744, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:49:04.101169 3649 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 20:49:04.101720 3649 webserver.cc:460] Webserver started at http://127.0.98.2:37131/ using document root <none> and password file <none>
I20260926 20:49:04.101974 3649 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:04.102058 3649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:04.102221 3649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:49:04.103354 3649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-1/data/instance:
uuid: "920586b4f2c741e880344195660ae334"
format_stamp: "Formatted at 2026-09-26 20:49:04 on dist-test-slave-hwlj"
server_key: "a58db61dd50a5db633af526163a449cb"
server_key_iv: "252f26fd75a8cea3e38f7ed0927b0cd6"
server_key_version: "encryptionkey@0"
I20260926 20:49:04.103817 3649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-1/wal/instance:
uuid: "920586b4f2c741e880344195660ae334"
format_stamp: "Formatted at 2026-09-26 20:49:04 on dist-test-slave-hwlj"
server_key: "a58db61dd50a5db633af526163a449cb"
server_key_iv: "252f26fd75a8cea3e38f7ed0927b0cd6"
server_key_version: "encryptionkey@0"
I20260926 20:49:04.105283 3649 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:49:04.106190 3664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:04.106392 3649 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:49:04.106495 3649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-1/wal
uuid: "920586b4f2c741e880344195660ae334"
format_stamp: "Formatted at 2026-09-26 20:49:04 on dist-test-slave-hwlj"
server_key: "a58db61dd50a5db633af526163a449cb"
server_key_iv: "252f26fd75a8cea3e38f7ed0927b0cd6"
server_key_version: "encryptionkey@0"
I20260926 20:49:04.106586 3649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:04.120042 3649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:04.122125 3649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:04.122293 3649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:04.122532 3649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:49:04.122848 3649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:49:04.122907 3649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:04.122951 3649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:49:04.122975 3649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:04.129168 3649 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45719
I20260926 20:49:04.129205 3777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45719 every 8 connection(s)
I20260926 20:49:04.129547 3649 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 20:49:04 dist-test-slave-hwlj krb5kdc[3408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455744, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:49:04.139017 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 3649
I20260926 20:49:04.139137 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-1/wal/instance
I20260926 20:49:04.139322 392 external_mini_cluster.cc:1428] Setting key 8fa79c37ff20779c1985784b498e63e1
I20260926 20:49:04.142190 3492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:04.130476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45231 (local address 127.0.98.62:43515)
0926 20:49:04.130520 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:04.130523 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:04.130643 (+ 120us) server_negotiation.cc:407] Connection header received
0926 20:49:04.131190 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:04.131192 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:04.131205 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:04.131311 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:04.132716 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:04.133313 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:04.134952 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:04.135272 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:04.135348 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:04.138365 (+ 3017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:04.138367 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:04.138367 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:04.138376 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:04.140664 (+ 2288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:04.141174 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:04.141175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:04.141175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:04.141230 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:04.141514 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:04.141514 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:04.141515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:04.141613 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:04.141714 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:04.141992 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 20:49:04.142082 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 20:49:04.142561 3780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:04.130351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43515 (local address 127.0.98.2:45231)
0926 20:49:04.130511 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:04.130553 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:04.131072 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:04.131412 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:04.131422 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:04.132319 (+ 897us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:04.132610 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:04.132617 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:04.133411 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:04.133413 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:04.134868 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:04.134871 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:04.135375 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:04.135377 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:04.135534 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:04.136189 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:04.136199 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:04.138270 (+ 2071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:04.140746 (+ 2476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:04.140754 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:04.140756 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:04.141095 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:04.141300 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:04.141301 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:04.141301 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:04.141449 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:04.141708 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:04.141714 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:04.141857 (+ 143us) client_negotiation.cc:769] Sending connection context
0926 20:49:04.141975 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 20:49:04.142150 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":92,"threads_started":1}
I20260926 20:49:04.143117 3778 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43515
I20260926 20:49:04.143215 3778 heartbeater.cc:464] Registering TS with master...
I20260926 20:49:04.143412 3778 heartbeater.cc:511] Master 127.0.98.62:43515 requested a full tablet report, sending...
I20260926 20:49:04.143855 3453 ts_manager.cc:194] Registered new tserver with Master: 920586b4f2c741e880344195660ae334 (127.0.98.2:45719)
I20260926 20:49:04.144232 3453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45231
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 20:49:04.168051 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43515
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:49:04.250099 3783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:04.250317 3783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:04.250342 3783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:04.251472 3783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:04.251511 3783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:04.251537 3783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:49:04.253047 3783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:04.253248 3783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:04.253430 3783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:04.258580 3788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:04.258713 3783 server_base.cc:1034] running on GCE node
W20260926 20:49:04.258569 3791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:04.258565 3789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:04.259093 3783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:04.259382 3783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:04.260542 3783 hybrid_clock.cc:648] HybridClock initialized: now 1790455744260476 us; error 83 us; skew 500 ppm
Sep 26 20:49:04 dist-test-slave-hwlj krb5kdc[3408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455744, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:49:04.263351 3783 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 20:49:04.263919 3783 webserver.cc:460] Webserver started at http://127.0.98.3:36893/ using document root <none> and password file <none>
I20260926 20:49:04.264173 3783 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:04.264220 3783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:04.264365 3783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:49:04.265440 3783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-2/data/instance:
uuid: "20d4488817b64193ad7e5498c049bfe3"
format_stamp: "Formatted at 2026-09-26 20:49:04 on dist-test-slave-hwlj"
server_key: "c1815ad0fd14e1b87bfab315aed639c7"
server_key_iv: "9768c328111cf133f2ed1c4640a3bcde"
server_key_version: "encryptionkey@0"
I20260926 20:49:04.265808 3783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-2/wal/instance:
uuid: "20d4488817b64193ad7e5498c049bfe3"
format_stamp: "Formatted at 2026-09-26 20:49:04 on dist-test-slave-hwlj"
server_key: "c1815ad0fd14e1b87bfab315aed639c7"
server_key_iv: "9768c328111cf133f2ed1c4640a3bcde"
server_key_version: "encryptionkey@0"
I20260926 20:49:04.267225 3783 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:49:04.268065 3798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:04.268242 3783 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:49:04.268345 3783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-2/wal
uuid: "20d4488817b64193ad7e5498c049bfe3"
format_stamp: "Formatted at 2026-09-26 20:49:04 on dist-test-slave-hwlj"
server_key: "c1815ad0fd14e1b87bfab315aed639c7"
server_key_iv: "9768c328111cf133f2ed1c4640a3bcde"
server_key_version: "encryptionkey@0"
I20260926 20:49:04.268428 3783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:04.282160 3783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:04.284672 3783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:04.284838 3783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:04.285074 3783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:49:04.285395 3783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:49:04.285454 3783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:04.285494 3783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:49:04.285522 3783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:04.291831 3783 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33585
I20260926 20:49:04.292021 3911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33585 every 8 connection(s)
I20260926 20:49:04.292268 3783 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
I20260926 20:49:04.293176 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 3783
I20260926 20:49:04.293274 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455725523895-392-0/minicluster-data/ts-2/wal/instance
I20260926 20:49:04.293473 392 external_mini_cluster.cc:1428] Setting key ebab70fad73ecb9251d0993f84fc13ed
Sep 26 20:49:04 dist-test-slave-hwlj krb5kdc[3408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455744, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:49:04.302259 3492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:04.292999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60001 (local address 127.0.98.62:43515)
0926 20:49:04.293077 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:04.293079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:04.293275 (+ 196us) server_negotiation.cc:407] Connection header received
0926 20:49:04.293995 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:04.293996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:04.294009 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:04.294073 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:04.295269 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:04.295935 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:04.297035 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:04.297226 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:04.297290 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:04.299414 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:04.299415 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:04.299416 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:04.299422 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:04.300967 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:04.301327 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:04.301328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:04.301328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:04.301377 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:04.301680 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:04.301681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:04.301682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:04.301805 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:04.301870 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:04.302107 (+ 237us) server_negotiation.cc:299] Negotiation successful
0926 20:49:04.302187 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 20:49:04.302394 3914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:04.292952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43515 (local address 127.0.98.3:60001)
0926 20:49:04.293153 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:04.293194 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:04.293852 (+ 658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:04.294150 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:04.294154 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:04.294912 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:04.295142 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:04.295147 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:04.296052 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:04.296053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:04.296871 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:04.296873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:04.297311 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:04.297312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:04.297419 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:04.297964 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:04.297972 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:04.299309 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:04.301041 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:04.301047 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:04.301048 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:04.301259 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:04.301435 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:04.301435 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:04.301436 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:04.301580 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:04.301873 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:04.301877 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:04.301993 (+ 116us) client_negotiation.cc:769] Sending connection context
0926 20:49:04.302070 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 20:49:04.302189 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":95,"threads_started":1}
I20260926 20:49:04.302908 3912 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43515
I20260926 20:49:04.302992 3912 heartbeater.cc:464] Registering TS with master...
I20260926 20:49:04.303215 3912 heartbeater.cc:511] Master 127.0.98.62:43515 requested a full tablet report, sending...
I20260926 20:49:04.303609 3453 ts_manager.cc:194] Registered new tserver with Master: 20d4488817b64193ad7e5498c049bfe3 (127.0.98.3:33585)
I20260926 20:49:04.304086 3453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60001
I20260926 20:49:04.305276 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 20:49:04 dist-test-slave-hwlj krb5kdc[3408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455744, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 20:49:04 dist-test-slave-hwlj krb5kdc[3408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455744, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:49:04.328097 3492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:04.320642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60508 (local address 127.0.98.62:43515)
0926 20:49:04.320696 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:04.320698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:04.320772 (+ 74us) server_negotiation.cc:407] Connection header received
0926 20:49:04.320838 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:04.320839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:04.320848 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:04.320898 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:04.321646 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:04.322126 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:04.322969 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:04.323147 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:04.323199 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:04.325785 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:04.325786 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:04.325786 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:04.325793 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:04.327030 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:04.327366 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:04.327366 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:04.327367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:04.327413 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:04.327596 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:04.327596 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:04.327596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:04.327738 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:04.327882 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:04.327976 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 20:49:04.328032 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260926 20:49:04.328380 3453 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:60508
I20260926 20:49:04.328902 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 3515
I20260926 20:49:04.333930 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 3649
I20260926 20:49:04.338660 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 3783
I20260926 20:49:04.342816 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 3421
2026-09-26T20:49:04Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1861 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:49:04 dist-test-slave-hwlj krb5kdc[3924](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:49:04 dist-test-slave-hwlj krb5kdc[3924](info): set up 2 sockets
Sep 26 20:49:04 dist-test-slave-hwlj krb5kdc[3924](info): commencing operation
krb5kdc: starting...
W20260926 20:49:05.387739 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/krb5kdc/test-user.keytab.
Sep 26 20:49:05 dist-test-slave-hwlj krb5kdc[3924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:49:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:49:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 20:49:05.507326 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36573
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36573
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:49:05.587773 3937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:05.588022 3937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:05.588061 3937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:05.589239 3937 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:49:05.589308 3937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:05.589332 3937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:05.589354 3937 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:49:05.589383 3937 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:49:05.590834 3937 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:05.591125 3937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:05.591356 3937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:05.596292 3943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:05.596307 3942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:05.596359 3945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:05.596541 3937 server_base.cc:1034] running on GCE node
I20260926 20:49:05.596782 3937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:05.597095 3937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:05.598244 3937 hybrid_clock.cc:648] HybridClock initialized: now 1790455745598232 us; error 33 us; skew 500 ppm
Sep 26 20:49:05 dist-test-slave-hwlj krb5kdc[3924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:49:05.600878 3937 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:49:05.601295 3937 webserver.cc:460] Webserver started at http://127.0.98.62:42577/ using document root <none> and password file <none>
I20260926 20:49:05.601480 3937 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:05.601522 3937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:05.601619 3937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:49:05.602648 3937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/master-0/data/instance:
uuid: "53dcf9711bab4f97b6550f34131c4f1c"
format_stamp: "Formatted at 2026-09-26 20:49:05 on dist-test-slave-hwlj"
server_key: "ecee04fdbec5c32fdf36dc43026d0611"
server_key_iv: "8167bf0e8ba3f47e09ddf9681cc237ae"
server_key_version: "encryptionkey@0"
I20260926 20:49:05.602949 3937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/master-0/wal/instance:
uuid: "53dcf9711bab4f97b6550f34131c4f1c"
format_stamp: "Formatted at 2026-09-26 20:49:05 on dist-test-slave-hwlj"
server_key: "ecee04fdbec5c32fdf36dc43026d0611"
server_key_iv: "8167bf0e8ba3f47e09ddf9681cc237ae"
server_key_version: "encryptionkey@0"
I20260926 20:49:05.604278 3937 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:49:05.604959 3952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:05.605135 3937 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 20:49:05.605238 3937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "53dcf9711bab4f97b6550f34131c4f1c"
format_stamp: "Formatted at 2026-09-26 20:49:05 on dist-test-slave-hwlj"
server_key: "ecee04fdbec5c32fdf36dc43026d0611"
server_key_iv: "8167bf0e8ba3f47e09ddf9681cc237ae"
server_key_version: "encryptionkey@0"
I20260926 20:49:05.605314 3937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:05.621929 3937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:05.624225 3937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:05.624387 3937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:05.628211 3937 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36573
I20260926 20:49:05.628242 4004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36573 every 8 connection(s)
I20260926 20:49:05.628585 3937 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:49:05.629402 4005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:49:05.631525 4005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53dcf9711bab4f97b6550f34131c4f1c: Bootstrap starting.
I20260926 20:49:05.632010 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 3937
I20260926 20:49:05.632128 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/master-0/wal/instance
I20260926 20:49:05.632208 4005 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53dcf9711bab4f97b6550f34131c4f1c: Neither blocks nor log segments found. Creating new log.
I20260926 20:49:05.632328 392 external_mini_cluster.cc:1428] Setting key c6c42ed794efe905f51cf66928472c3b
I20260926 20:49:05.632531 4005 log.cc:824] T 00000000000000000000000000000000 P 53dcf9711bab4f97b6550f34131c4f1c: Log is configured to *not* fsync() on all Append() calls
I20260926 20:49:05.633229 4005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53dcf9711bab4f97b6550f34131c4f1c: No bootstrap required, opened a new log
I20260926 20:49:05.635025 4005 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53dcf9711bab4f97b6550f34131c4f1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53dcf9711bab4f97b6550f34131c4f1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36573 } }
I20260926 20:49:05.635164 4005 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53dcf9711bab4f97b6550f34131c4f1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:49:05.635186 4005 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53dcf9711bab4f97b6550f34131c4f1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53dcf9711bab4f97b6550f34131c4f1c, State: Initialized, Role: FOLLOWER
I20260926 20:49:05.635318 4005 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53dcf9711bab4f97b6550f34131c4f1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53dcf9711bab4f97b6550f34131c4f1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36573 } }
I20260926 20:49:05.635401 4005 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53dcf9711bab4f97b6550f34131c4f1c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:49:05.635454 4005 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53dcf9711bab4f97b6550f34131c4f1c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:49:05.635502 4005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53dcf9711bab4f97b6550f34131c4f1c [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:49:05.636097 4005 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53dcf9711bab4f97b6550f34131c4f1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53dcf9711bab4f97b6550f34131c4f1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36573 } }
I20260926 20:49:05.636214 4005 leader_election.cc:302] T 00000000000000000000000000000000 P 53dcf9711bab4f97b6550f34131c4f1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53dcf9711bab4f97b6550f34131c4f1c; no voters:
I20260926 20:49:05.636381 4005 leader_election.cc:288] T 00000000000000000000000000000000 P 53dcf9711bab4f97b6550f34131c4f1c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:49:05.636416 4010 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53dcf9711bab4f97b6550f34131c4f1c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:49:05.636595 4010 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53dcf9711bab4f97b6550f34131c4f1c [term 1 LEADER]: Becoming Leader. State: Replica: 53dcf9711bab4f97b6550f34131c4f1c, State: Running, Role: LEADER
I20260926 20:49:05.636654 4005 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53dcf9711bab4f97b6550f34131c4f1c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:49:05.636731 4010 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53dcf9711bab4f97b6550f34131c4f1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53dcf9711bab4f97b6550f34131c4f1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36573 } }
I20260926 20:49:05.638406 4012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53dcf9711bab4f97b6550f34131c4f1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53dcf9711bab4f97b6550f34131c4f1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53dcf9711bab4f97b6550f34131c4f1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36573 } } }
I20260926 20:49:05.638816 4012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53dcf9711bab4f97b6550f34131c4f1c [sys.catalog]: This master's current role is: LEADER
I20260926 20:49:05.638406 4011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53dcf9711bab4f97b6550f34131c4f1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53dcf9711bab4f97b6550f34131c4f1c. Latest consensus state: current_term: 1 leader_uuid: "53dcf9711bab4f97b6550f34131c4f1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53dcf9711bab4f97b6550f34131c4f1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36573 } } }
I20260926 20:49:05.638988 4011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53dcf9711bab4f97b6550f34131c4f1c [sys.catalog]: This master's current role is: LEADER
W20260926 20:49:05.638556 4025 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 53dcf9711bab4f97b6550f34131c4f1c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:49:05.639078 4025 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 26 20:49:05 dist-test-slave-hwlj krb5kdc[3924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:49:05.639886 4027 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:49:05.640138 4027 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:49:05.641865 4027 catalog_manager.cc:1347] Generated new cluster ID: 7509b7bbf37e4ceebe177d3d8d7de540
I20260926 20:49:05.641960 4027 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:49:05.643483 4009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:05.632817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49644 (local address 127.0.98.62:36573)
0926 20:49:05.632968 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:05.632978 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:05.633010 (+ 32us) server_negotiation.cc:407] Connection header received
0926 20:49:05.633512 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:05.633519 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:05.633567 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:05.633712 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:05.634321 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:05.635211 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:05.636123 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:05.636767 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:05.636872 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:05.639822 (+ 2950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:05.639826 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:05.639827 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:05.639839 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:05.641851 (+ 2012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:05.642252 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:05.642255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:05.642255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:05.642351 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:05.642565 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:05.642565 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:05.642566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:05.642844 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:05.642980 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:05.643114 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 20:49:05.643239 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":67,"threads_started":1}
I20260926 20:49:05.660115 4027 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:49:05.660869 4027 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:49:05.663793 4027 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53dcf9711bab4f97b6550f34131c4f1c: Generated new TSK 0
I20260926 20:49:05.663990 4027 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 20:49:05.694345 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36573
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:49:05.775271 4031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:05.775506 4031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:05.775547 4031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:05.776731 4031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:05.776798 4031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:05.776844 4031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:49:05.778333 4031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:05.778596 4031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:05.778842 4031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:05.783780 4039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:05.783792 4036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:05.783847 4031 server_base.cc:1034] running on GCE node
W20260926 20:49:05.783799 4037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:05.784273 4031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:05.784516 4031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:05.785657 4031 hybrid_clock.cc:648] HybridClock initialized: now 1790455745785649 us; error 29 us; skew 500 ppm
Sep 26 20:49:05 dist-test-slave-hwlj krb5kdc[3924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:49:05.788103 4031 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 20:49:05.788513 4031 webserver.cc:460] Webserver started at http://127.0.98.1:37079/ using document root <none> and password file <none>
I20260926 20:49:05.788731 4031 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:05.788775 4031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:05.788913 4031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:49:05.790071 4031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/ts-0/data/instance:
uuid: "5d3fca75dabb492dbda85e6f14842517"
format_stamp: "Formatted at 2026-09-26 20:49:05 on dist-test-slave-hwlj"
server_key: "28beefa51c72be29a495f03664d845c5"
server_key_iv: "e9eb30d2bb33fb34efbc8b797b483264"
server_key_version: "encryptionkey@0"
I20260926 20:49:05.790511 4031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5d3fca75dabb492dbda85e6f14842517"
format_stamp: "Formatted at 2026-09-26 20:49:05 on dist-test-slave-hwlj"
server_key: "28beefa51c72be29a495f03664d845c5"
server_key_iv: "e9eb30d2bb33fb34efbc8b797b483264"
server_key_version: "encryptionkey@0"
I20260926 20:49:05.791918 4031 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:49:05.792662 4046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:05.792855 4031 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:49:05.792950 4031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/ts-0/wal
uuid: "5d3fca75dabb492dbda85e6f14842517"
format_stamp: "Formatted at 2026-09-26 20:49:05 on dist-test-slave-hwlj"
server_key: "28beefa51c72be29a495f03664d845c5"
server_key_iv: "e9eb30d2bb33fb34efbc8b797b483264"
server_key_version: "encryptionkey@0"
I20260926 20:49:05.793038 4031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:05.810335 4031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:05.812252 4031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:05.812414 4031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:05.812657 4031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:49:05.812956 4031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:49:05.813009 4031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:05.813042 4031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:49:05.813073 4031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:05.818614 4031 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38359
I20260926 20:49:05.818655 4159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38359 every 8 connection(s)
I20260926 20:49:05.819000 4031 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:49:05.819461 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 4031
I20260926 20:49:05.819577 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/ts-0/wal/instance
I20260926 20:49:05.819813 392 external_mini_cluster.cc:1428] Setting key 0294c58f365894038ebfda1c4ef26fef
Sep 26 20:49:05 dist-test-slave-hwlj krb5kdc[3924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 20:49:05.830021 4009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:05.819735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49553 (local address 127.0.98.62:36573)
0926 20:49:05.819810 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:05.819812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:05.820080 (+ 268us) server_negotiation.cc:407] Connection header received
0926 20:49:05.821371 (+ 1291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:05.821372 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:05.821382 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:05.821424 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:05.822293 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:05.822796 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:05.824415 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:05.824622 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:05.824681 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:05.827145 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:05.827147 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:05.827147 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:05.827154 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:05.828693 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:05.829096 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:05.829098 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:05.829098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:05.829156 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:05.829403 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:05.829404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:05.829405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:05.829574 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:05.829643 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:05.829887 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 20:49:05.829956 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 20:49:05.830189 4162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:05.819726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36573 (local address 127.0.98.1:49553)
0926 20:49:05.819926 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:05.819975 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:05.821263 (+ 1288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:05.821502 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:05.821507 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:05.821913 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:05.822182 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:05.822187 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:05.822892 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:05.822893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:05.824324 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:05.824327 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:05.824712 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:05.824713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:05.824865 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:05.825397 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:05.825409 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:05.827046 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:05.828780 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:05.828786 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:05.828788 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:05.829005 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:05.829223 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:05.829224 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:05.829224 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:05.829327 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:05.829643 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:05.829648 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:05.829767 (+ 119us) client_negotiation.cc:769] Sending connection context
0926 20:49:05.829859 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 20:49:05.829977 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":92,"threads_started":1}
I20260926 20:49:05.830679 4160 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36573
I20260926 20:49:05.830762 4160 heartbeater.cc:464] Registering TS with master...
I20260926 20:49:05.830900 4160 heartbeater.cc:511] Master 127.0.98.62:36573 requested a full tablet report, sending...
I20260926 20:49:05.831393 3969 ts_manager.cc:194] Registered new tserver with Master: 5d3fca75dabb492dbda85e6f14842517 (127.0.98.1:38359)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 20:49:05.832259 3969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49553
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 20:49:05.847239 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36573
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:49:05.927400 4165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:05.927651 4165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:05.927688 4165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:05.928907 4165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:05.928977 4165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:05.929023 4165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:49:05.930478 4165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:05.930757 4165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:05.930981 4165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:05.935984 4173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:05.935992 4170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:05.935992 4171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:05.936000 4165 server_base.cc:1034] running on GCE node
I20260926 20:49:05.936422 4165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:05.936668 4165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:05.937812 4165 hybrid_clock.cc:648] HybridClock initialized: now 1790455745937782 us; error 61 us; skew 500 ppm
Sep 26 20:49:05 dist-test-slave-hwlj krb5kdc[3924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:49:05.940372 4165 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 20:49:05.940804 4165 webserver.cc:460] Webserver started at http://127.0.98.2:44969/ using document root <none> and password file <none>
I20260926 20:49:05.941052 4165 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:05.941100 4165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:05.941258 4165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:49:05.942276 4165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/ts-1/data/instance:
uuid: "976c0971ae1b4d3792079c1d8991f384"
format_stamp: "Formatted at 2026-09-26 20:49:05 on dist-test-slave-hwlj"
server_key: "d383ec07fa99608ca5c831fb58765a29"
server_key_iv: "0b58080bced24d572dd85c2c33b44349"
server_key_version: "encryptionkey@0"
I20260926 20:49:05.942625 4165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/ts-1/wal/instance:
uuid: "976c0971ae1b4d3792079c1d8991f384"
format_stamp: "Formatted at 2026-09-26 20:49:05 on dist-test-slave-hwlj"
server_key: "d383ec07fa99608ca5c831fb58765a29"
server_key_iv: "0b58080bced24d572dd85c2c33b44349"
server_key_version: "encryptionkey@0"
I20260926 20:49:05.943933 4165 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:49:05.944602 4180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:05.944770 4165 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:49:05.944865 4165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/ts-1/wal
uuid: "976c0971ae1b4d3792079c1d8991f384"
format_stamp: "Formatted at 2026-09-26 20:49:05 on dist-test-slave-hwlj"
server_key: "d383ec07fa99608ca5c831fb58765a29"
server_key_iv: "0b58080bced24d572dd85c2c33b44349"
server_key_version: "encryptionkey@0"
I20260926 20:49:05.944942 4165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:05.959892 4165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:05.962325 4165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:05.962498 4165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:05.962739 4165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:49:05.963086 4165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:49:05.963137 4165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:05.963177 4165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:49:05.963207 4165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:05.969154 4165 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37383
I20260926 20:49:05.969197 4293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37383 every 8 connection(s)
I20260926 20:49:05.969528 4165 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
I20260926 20:49:05.971815 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 4165
I20260926 20:49:05.971942 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/ts-1/wal/instance
I20260926 20:49:05.972110 392 external_mini_cluster.cc:1428] Setting key f9a9c62dd0b34aa68fe21bd1725c7003
Sep 26 20:49:05 dist-test-slave-hwlj krb5kdc[3924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:49:05.979912 4009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:05.970339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37467 (local address 127.0.98.62:36573)
0926 20:49:05.970410 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:05.970412 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:05.970644 (+ 232us) server_negotiation.cc:407] Connection header received
0926 20:49:05.971552 (+ 908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:05.971553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:05.971564 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:05.971648 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:05.972411 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:05.973102 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:05.974129 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:05.974333 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:05.974419 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:05.976957 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:05.976959 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:05.976959 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:05.976965 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:05.978363 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:05.978885 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:05.978886 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:05.978886 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:05.978935 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:05.979215 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:05.979216 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:05.979216 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:05.979316 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:05.979366 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:05.979722 (+ 356us) server_negotiation.cc:299] Negotiation successful
0926 20:49:05.979846 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 20:49:05.980194 4296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:05.970320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36573 (local address 127.0.98.2:37467)
0926 20:49:05.970524 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:05.970560 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:05.971450 (+ 890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:05.971705 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:05.971709 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:05.972100 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:05.972345 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:05.972349 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:05.973197 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:05.973198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:05.974012 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:05.974014 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:05.974425 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:05.974427 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:05.974576 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:05.975115 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:05.975124 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:05.976865 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:05.978451 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:05.978459 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:05.978461 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:05.978798 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:05.978998 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:05.978998 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:05.978999 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:05.979149 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:05.979380 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:05.979386 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:05.979573 (+ 187us) client_negotiation.cc:769] Sending connection context
0926 20:49:05.979704 (+ 131us) client_negotiation.cc:240] Negotiation successful
0926 20:49:05.979919 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":109,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 20:49:05.980772 4294 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36573
I20260926 20:49:05.980895 4294 heartbeater.cc:464] Registering TS with master...
I20260926 20:49:05.981077 4294 heartbeater.cc:511] Master 127.0.98.62:36573 requested a full tablet report, sending...
I20260926 20:49:05.981448 3969 ts_manager.cc:194] Registered new tserver with Master: 976c0971ae1b4d3792079c1d8991f384 (127.0.98.2:37383)
I20260926 20:49:05.981844 3969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37467
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 20:49:05.999701 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36573
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:49:06.078855 4299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:06.079066 4299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:06.079088 4299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:06.080214 4299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:06.080250 4299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:06.080276 4299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:49:06.081683 4299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:06.081867 4299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:06.082038 4299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:49:06.086997 4299 server_base.cc:1034] running on GCE node
W20260926 20:49:06.086970 4304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:06.086954 4307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:06.086958 4305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:06.087437 4299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:06.087677 4299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:06.088822 4299 hybrid_clock.cc:648] HybridClock initialized: now 1790455746088820 us; error 36 us; skew 500 ppm
Sep 26 20:49:06 dist-test-slave-hwlj krb5kdc[3924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:49:06.091295 4299 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 20:49:06.091800 4299 webserver.cc:460] Webserver started at http://127.0.98.3:41899/ using document root <none> and password file <none>
I20260926 20:49:06.092000 4299 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:06.092042 4299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:06.092130 4299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:49:06.093088 4299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/ts-2/data/instance:
uuid: "aae349f11525462fb1bf5ea0da03a45c"
format_stamp: "Formatted at 2026-09-26 20:49:06 on dist-test-slave-hwlj"
server_key: "cd06b63fdd990d870d301cc7f3ca10f6"
server_key_iv: "02d13216dcadc19bc072ce37aeef6877"
server_key_version: "encryptionkey@0"
I20260926 20:49:06.093374 4299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/ts-2/wal/instance:
uuid: "aae349f11525462fb1bf5ea0da03a45c"
format_stamp: "Formatted at 2026-09-26 20:49:06 on dist-test-slave-hwlj"
server_key: "cd06b63fdd990d870d301cc7f3ca10f6"
server_key_iv: "02d13216dcadc19bc072ce37aeef6877"
server_key_version: "encryptionkey@0"
I20260926 20:49:06.094578 4299 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:49:06.095234 4314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:06.095384 4299 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:49:06.095463 4299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/ts-2/wal
uuid: "aae349f11525462fb1bf5ea0da03a45c"
format_stamp: "Formatted at 2026-09-26 20:49:06 on dist-test-slave-hwlj"
server_key: "cd06b63fdd990d870d301cc7f3ca10f6"
server_key_iv: "02d13216dcadc19bc072ce37aeef6877"
server_key_version: "encryptionkey@0"
I20260926 20:49:06.095526 4299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:06.105028 4299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:06.106773 4299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:06.106879 4299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:06.107039 4299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:49:06.107292 4299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:49:06.107362 4299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:06.107398 4299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:49:06.107446 4299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:06.113158 4299 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37161
I20260926 20:49:06.113229 4427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37161 every 8 connection(s)
I20260926 20:49:06.113637 4299 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
I20260926 20:49:06.114293 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 4299
I20260926 20:49:06.114413 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455725523895-392-0/minicluster-data/ts-2/wal/instance
I20260926 20:49:06.114629 392 external_mini_cluster.cc:1428] Setting key e72c9c15f7b327ad271a36edd9e03adc
Sep 26 20:49:06 dist-test-slave-hwlj krb5kdc[3924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:49:06.123613 4009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:06.114454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52771 (local address 127.0.98.62:36573)
0926 20:49:06.114522 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:06.114525 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:06.114739 (+ 214us) server_negotiation.cc:407] Connection header received
0926 20:49:06.116035 (+ 1296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:06.116036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:06.116048 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:06.116090 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:06.116713 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:06.117215 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:06.118269 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:06.118495 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:06.118576 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:06.120776 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:06.120777 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:06.120778 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:06.120786 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:06.122185 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:06.122639 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:06.122641 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:06.122642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:06.122700 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:06.122974 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:06.122975 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:06.122975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:06.123091 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:06.123168 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:06.123434 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 20:49:06.123513 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 20:49:06.123858 4430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:06.114410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36573 (local address 127.0.98.3:52771)
0926 20:49:06.114622 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:06.114662 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:06.115913 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:06.116169 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:06.116180 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:06.116435 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:06.116622 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:06.116626 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:06.117355 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:06.117356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:06.118175 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:06.118177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:06.118582 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:06.118583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:06.118718 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:06.119265 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:06.119273 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:06.120671 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:06.122302 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:06.122309 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:06.122317 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:06.122546 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:06.122777 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:06.122778 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:06.122778 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:06.122889 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:06.123160 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:06.123166 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:06.123300 (+ 134us) client_negotiation.cc:769] Sending connection context
0926 20:49:06.123416 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 20:49:06.123589 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":87,"threads_started":1}
I20260926 20:49:06.124325 4428 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36573
I20260926 20:49:06.124437 4428 heartbeater.cc:464] Registering TS with master...
I20260926 20:49:06.124627 4428 heartbeater.cc:511] Master 127.0.98.62:36573 requested a full tablet report, sending...
I20260926 20:49:06.124974 3969 ts_manager.cc:194] Registered new tserver with Master: aae349f11525462fb1bf5ea0da03a45c (127.0.98.3:37161)
I20260926 20:49:06.125442 3969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52771
I20260926 20:49:06.126327 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 20:49:06 dist-test-slave-hwlj krb5kdc[3924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 20:49:06 dist-test-slave-hwlj krb5kdc[3924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 20:49:06.150331 4435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:06.142046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59052 (local address 127.0.98.1:38359)
0926 20:49:06.142237 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:06.142245 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:06.142254 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:49:06.142279 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:06.142281 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:06.142296 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:06.142402 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:06.142994 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:06.143731 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:06.144498 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:06.144883 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:06.144979 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:06.147351 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:06.147361 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:06.147361 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:06.147369 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:06.149052 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:06.149420 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:06.149424 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:06.149424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:06.149511 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:06.149729 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:06.149730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:06.149730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:06.149869 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:06.150028 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:06.150118 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 20:49:06.150201 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":69,"threads_started":1}
Sep 26 20:49:06 dist-test-slave-hwlj krb5kdc[3924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:49:06.159494 4009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:06.151563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49648 (local address 127.0.98.62:36573)
0926 20:49:06.151627 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:06.151629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:06.151665 (+ 36us) server_negotiation.cc:407] Connection header received
0926 20:49:06.151798 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:06.151799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:06.151810 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:06.151867 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:06.152351 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:06.152847 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:06.153563 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:06.153774 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:06.153858 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:06.155918 (+ 2060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:06.155919 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:06.155919 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:06.155927 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:06.157069 (+ 1142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:06.157406 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:06.157407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:06.157407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:06.157470 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:06.157679 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:06.157680 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:06.157680 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:06.157835 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:06.157959 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:06.159306 (+ 1347us) server_negotiation.cc:299] Negotiation successful
0926 20:49:06.159425 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260926 20:49:06.159803 3969 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49648
I20260926 20:49:06.160175 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 4031
I20260926 20:49:06.165220 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 4165
I20260926 20:49:06.169797 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 4299
I20260926 20:49:06.173969 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 3937
2026-09-26T20:49:06Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1830 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:49:06 dist-test-slave-hwlj krb5kdc[4440](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:49:06 dist-test-slave-hwlj krb5kdc[4440](info): set up 2 sockets
Sep 26 20:49:06 dist-test-slave-hwlj krb5kdc[4440](info): commencing operation
krb5kdc: starting...
W20260926 20:49:07.242640 392 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260926 20:49:07.423516 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.181s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/krb5kdc/test-user.keytab.
Sep 26 20:49:07 dist-test-slave-hwlj krb5kdc[4440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:49:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:49:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 20:49:07.539245 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40229
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:49:07.618770 4453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:07.619024 4453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:07.619064 4453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:07.620255 4453 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:49:07.620321 4453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:07.620344 4453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:07.620364 4453 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:49:07.620395 4453 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:49:07.621858 4453 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:07.622133 4453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:07.622356 4453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:07.627305 4461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:07.627315 4459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:07.627331 4458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:07.627661 4453 server_base.cc:1034] running on GCE node
I20260926 20:49:07.627966 4453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:07.628253 4453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:07.629420 4453 hybrid_clock.cc:648] HybridClock initialized: now 1790455747629392 us; error 47 us; skew 500 ppm
Sep 26 20:49:07 dist-test-slave-hwlj krb5kdc[4440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455747, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:49:07.632001 4453 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:49:07.632376 4453 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:07.632447 4453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:07.632584 4453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:49:07.633700 4453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/master-0/data/instance:
uuid: "75043a94e3474209977d45b6f4463bea"
format_stamp: "Formatted at 2026-09-26 20:49:07 on dist-test-slave-hwlj"
server_key: "5b15a494bd5aa5136ac2d6f86090ae76"
server_key_iv: "9be3a9f695d48b680301ca729dc4c5b8"
server_key_version: "encryptionkey@0"
I20260926 20:49:07.634090 4453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/master-0/wal/instance:
uuid: "75043a94e3474209977d45b6f4463bea"
format_stamp: "Formatted at 2026-09-26 20:49:07 on dist-test-slave-hwlj"
server_key: "5b15a494bd5aa5136ac2d6f86090ae76"
server_key_iv: "9be3a9f695d48b680301ca729dc4c5b8"
server_key_version: "encryptionkey@0"
I20260926 20:49:07.635397 4453 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:49:07.636147 4467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:07.636302 4453 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:07.636406 4453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "75043a94e3474209977d45b6f4463bea"
format_stamp: "Formatted at 2026-09-26 20:49:07 on dist-test-slave-hwlj"
server_key: "5b15a494bd5aa5136ac2d6f86090ae76"
server_key_iv: "9be3a9f695d48b680301ca729dc4c5b8"
server_key_version: "encryptionkey@0"
I20260926 20:49:07.636479 4453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:07.646469 4453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:07.648690 4453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:07.648862 4453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:07.652825 4453 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40229
I20260926 20:49:07.652855 4519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40229 every 8 connection(s)
I20260926 20:49:07.653146 4453 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:49:07.654109 4520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:49:07.654099 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 4453
I20260926 20:49:07.654244 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/master-0/wal/instance
I20260926 20:49:07.654433 392 external_mini_cluster.cc:1428] Setting key 713f8ebe97708f3940e8fcd24aba845c
I20260926 20:49:07.657505 4520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75043a94e3474209977d45b6f4463bea: Bootstrap starting.
I20260926 20:49:07.658162 4520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75043a94e3474209977d45b6f4463bea: Neither blocks nor log segments found. Creating new log.
I20260926 20:49:07.658510 4520 log.cc:824] T 00000000000000000000000000000000 P 75043a94e3474209977d45b6f4463bea: Log is configured to *not* fsync() on all Append() calls
I20260926 20:49:07.659240 4520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75043a94e3474209977d45b6f4463bea: No bootstrap required, opened a new log
Sep 26 20:49:07 dist-test-slave-hwlj krb5kdc[4440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:49:07.661444 4520 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75043a94e3474209977d45b6f4463bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75043a94e3474209977d45b6f4463bea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40229 } }
I20260926 20:49:07.661633 4520 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75043a94e3474209977d45b6f4463bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:49:07.661688 4520 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75043a94e3474209977d45b6f4463bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75043a94e3474209977d45b6f4463bea, State: Initialized, Role: FOLLOWER
I20260926 20:49:07.661847 4520 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75043a94e3474209977d45b6f4463bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75043a94e3474209977d45b6f4463bea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40229 } }
I20260926 20:49:07.661947 4520 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75043a94e3474209977d45b6f4463bea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:49:07.661985 4520 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75043a94e3474209977d45b6f4463bea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:49:07.662051 4520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75043a94e3474209977d45b6f4463bea [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:49:07.662786 4520 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75043a94e3474209977d45b6f4463bea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75043a94e3474209977d45b6f4463bea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40229 } }
I20260926 20:49:07.662904 4520 leader_election.cc:302] T 00000000000000000000000000000000 P 75043a94e3474209977d45b6f4463bea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75043a94e3474209977d45b6f4463bea; no voters:
I20260926 20:49:07.663074 4520 leader_election.cc:288] T 00000000000000000000000000000000 P 75043a94e3474209977d45b6f4463bea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:49:07.663162 4525 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75043a94e3474209977d45b6f4463bea [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:49:07.663329 4520 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75043a94e3474209977d45b6f4463bea [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:49:07.663352 4525 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75043a94e3474209977d45b6f4463bea [term 1 LEADER]: Becoming Leader. State: Replica: 75043a94e3474209977d45b6f4463bea, State: Running, Role: LEADER
I20260926 20:49:07.663893 4525 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75043a94e3474209977d45b6f4463bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75043a94e3474209977d45b6f4463bea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40229 } }
I20260926 20:49:07.664582 4526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75043a94e3474209977d45b6f4463bea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75043a94e3474209977d45b6f4463bea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75043a94e3474209977d45b6f4463bea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40229 } } }
I20260926 20:49:07.664661 4526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75043a94e3474209977d45b6f4463bea [sys.catalog]: This master's current role is: LEADER
I20260926 20:49:07.665060 4527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75043a94e3474209977d45b6f4463bea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75043a94e3474209977d45b6f4463bea. Latest consensus state: current_term: 1 leader_uuid: "75043a94e3474209977d45b6f4463bea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75043a94e3474209977d45b6f4463bea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40229 } } }
I20260926 20:49:07.665175 4527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75043a94e3474209977d45b6f4463bea [sys.catalog]: This master's current role is: LEADER
W20260926 20:49:07.665252 4539 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 75043a94e3474209977d45b6f4463bea: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:49:07.665295 4539 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:49:07.665381 4540 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:49:07.665624 4540 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:49:07.665860 4523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:07.654968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40376 (local address 127.0.98.62:40229)
0926 20:49:07.655137 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:07.655145 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:07.655169 (+ 24us) server_negotiation.cc:407] Connection header received
0926 20:49:07.655621 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:07.655628 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:07.655676 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:07.655864 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:07.656678 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:07.657533 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:07.658347 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:07.658941 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:07.659053 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:07.661749 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:07.661753 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:07.661753 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:07.661766 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:07.663688 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:07.664151 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:07.664155 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:07.664156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:07.664273 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:07.664529 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:07.664530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:07.664531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:07.664876 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:07.665031 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:07.665511 (+ 480us) server_negotiation.cc:299] Negotiation successful
0926 20:49:07.665649 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":83,"threads_started":1}
I20260926 20:49:07.667109 4540 catalog_manager.cc:1347] Generated new cluster ID: 0f7a7b7ce2f047ffb692ca71f2d8a338
I20260926 20:49:07.667156 4540 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:49:07.687150 4540 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:49:07.687727 4540 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:49:07.691026 4540 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75043a94e3474209977d45b6f4463bea: Generated new TSK 0
I20260926 20:49:07.691226 4540 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 20:49:07.719230 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40229
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:49:07.806859 4546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:07.807092 4546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:07.807133 4546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:07.808336 4546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:07.808404 4546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:07.808451 4546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:49:07.809989 4546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:07.810250 4546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:07.810483 4546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:07.815596 4554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:07.815608 4551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:07.815636 4552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:07.815873 4546 server_base.cc:1034] running on GCE node
I20260926 20:49:07.816128 4546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:07.816396 4546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:07.817551 4546 hybrid_clock.cc:648] HybridClock initialized: now 1790455747817530 us; error 36 us; skew 500 ppm
Sep 26 20:49:07 dist-test-slave-hwlj krb5kdc[4440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455747, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:49:07.820307 4546 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 20:49:07.820755 4546 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:07.820830 4546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:07.820969 4546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:49:07.822072 4546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/ts-0/data/instance:
uuid: "f0c9456ff6084ab98db1d8c656ba2ad9"
format_stamp: "Formatted at 2026-09-26 20:49:07 on dist-test-slave-hwlj"
server_key: "4624334f9866c28300b142e20606341a"
server_key_iv: "4e675da8a5f23d5f4da9aa04f5a0bcad"
server_key_version: "encryptionkey@0"
I20260926 20:49:07.822460 4546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f0c9456ff6084ab98db1d8c656ba2ad9"
format_stamp: "Formatted at 2026-09-26 20:49:07 on dist-test-slave-hwlj"
server_key: "4624334f9866c28300b142e20606341a"
server_key_iv: "4e675da8a5f23d5f4da9aa04f5a0bcad"
server_key_version: "encryptionkey@0"
I20260926 20:49:07.823837 4546 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 20:49:07.824810 4560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:07.825053 4546 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:49:07.825155 4546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/ts-0/wal
uuid: "f0c9456ff6084ab98db1d8c656ba2ad9"
format_stamp: "Formatted at 2026-09-26 20:49:07 on dist-test-slave-hwlj"
server_key: "4624334f9866c28300b142e20606341a"
server_key_iv: "4e675da8a5f23d5f4da9aa04f5a0bcad"
server_key_version: "encryptionkey@0"
I20260926 20:49:07.825227 4546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:07.842756 4546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:07.844745 4546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:07.844861 4546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:07.845048 4546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:49:07.845369 4546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:49:07.845404 4546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:07.845453 4546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:49:07.845479 4546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:07.851191 4546 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43047
I20260926 20:49:07.851262 4673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43047 every 8 connection(s)
I20260926 20:49:07.851555 4546 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:49:07.854387 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 4546
I20260926 20:49:07.854498 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/ts-0/wal/instance
I20260926 20:49:07.854687 392 external_mini_cluster.cc:1428] Setting key 6c0e1965b24ce8a92a9b68c82c2c1e30
Sep 26 20:49:07 dist-test-slave-hwlj krb5kdc[4440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455747, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:49:07.861123 4523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:07.852288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41151 (local address 127.0.98.62:40229)
0926 20:49:07.852330 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:07.852332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:07.852548 (+ 216us) server_negotiation.cc:407] Connection header received
0926 20:49:07.852992 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:07.852994 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:07.853004 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:07.853054 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:07.853921 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:07.854570 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:07.855583 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:07.855937 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:07.856011 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:07.858310 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:07.858311 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:07.858312 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:07.858319 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:07.859708 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:07.860170 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:07.860171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:07.860172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:07.860231 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:07.860491 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:07.860492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:07.860492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:07.860602 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:07.860678 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:07.860952 (+ 274us) server_negotiation.cc:299] Negotiation successful
0926 20:49:07.861035 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 20:49:07.861318 4676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:07.852281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40229 (local address 127.0.98.1:41151)
0926 20:49:07.852425 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:07.852459 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:07.852910 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:07.853108 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:07.853111 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:07.853363 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:07.853842 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:07.853847 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:07.854657 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:07.854658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:07.855496 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:07.855498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:07.856015 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:07.856016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:07.856115 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:07.856666 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:07.856677 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:07.858206 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:07.859867 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:07.859874 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:07.859876 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:07.860096 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:07.860297 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:07.860298 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:07.860299 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:07.860407 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:07.860669 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:07.860675 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:07.860811 (+ 136us) client_negotiation.cc:769] Sending connection context
0926 20:49:07.860925 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 20:49:07.861081 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":90,"threads_started":1}
I20260926 20:49:07.861857 4674 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40229
I20260926 20:49:07.861967 4674 heartbeater.cc:464] Registering TS with master...
I20260926 20:49:07.862136 4674 heartbeater.cc:511] Master 127.0.98.62:40229 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 20:49:07.862675 4484 ts_manager.cc:194] Registered new tserver with Master: f0c9456ff6084ab98db1d8c656ba2ad9 (127.0.98.1:43047)
I20260926 20:49:07.863276 4484 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41151
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 20:49:07.879892 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40229
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:49:07.959992 4679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:07.960206 4679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:07.960229 4679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:07.961386 4679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:07.961429 4679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:07.961455 4679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:49:07.963011 4679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:07.963243 4679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:07.963513 4679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:07.968504 4685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:07.968497 4684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:07.968497 4687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:07.968799 4679 server_base.cc:1034] running on GCE node
I20260926 20:49:07.969054 4679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:07.969300 4679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:07.970469 4679 hybrid_clock.cc:648] HybridClock initialized: now 1790455747970465 us; error 35 us; skew 500 ppm
Sep 26 20:49:07 dist-test-slave-hwlj krb5kdc[4440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455747, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:49:07.973032 4679 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 20:49:07.973313 4679 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:07.973356 4679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:07.973453 4679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:49:07.974459 4679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/ts-1/data/instance:
uuid: "121b1093d17048f7a64b641f4ec5e9dd"
format_stamp: "Formatted at 2026-09-26 20:49:07 on dist-test-slave-hwlj"
server_key: "d3b1cdb74091261a1e2908d1a1b03046"
server_key_iv: "0642e36d2382ccd8548f537d7e6b63ad"
server_key_version: "encryptionkey@0"
I20260926 20:49:07.974757 4679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/ts-1/wal/instance:
uuid: "121b1093d17048f7a64b641f4ec5e9dd"
format_stamp: "Formatted at 2026-09-26 20:49:07 on dist-test-slave-hwlj"
server_key: "d3b1cdb74091261a1e2908d1a1b03046"
server_key_iv: "0642e36d2382ccd8548f537d7e6b63ad"
server_key_version: "encryptionkey@0"
I20260926 20:49:07.976089 4679 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:49:07.976771 4693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:07.976979 4679 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:49:07.977068 4679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/ts-1/wal
uuid: "121b1093d17048f7a64b641f4ec5e9dd"
format_stamp: "Formatted at 2026-09-26 20:49:07 on dist-test-slave-hwlj"
server_key: "d3b1cdb74091261a1e2908d1a1b03046"
server_key_iv: "0642e36d2382ccd8548f537d7e6b63ad"
server_key_version: "encryptionkey@0"
I20260926 20:49:07.977139 4679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:07.998520 4679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:08.001044 4679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:08.001196 4679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:08.001446 4679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:49:08.001766 4679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:49:08.001801 4679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:08.001822 4679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:49:08.001835 4679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:08.007464 4679 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37797
I20260926 20:49:08.007498 4806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37797 every 8 connection(s)
I20260926 20:49:08.007786 4679 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 20:49:08 dist-test-slave-hwlj krb5kdc[4440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455747, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:49:08.014628 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 4679
I20260926 20:49:08.014751 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/ts-1/wal/instance
I20260926 20:49:08.014988 392 external_mini_cluster.cc:1428] Setting key f99be79d6abb0c30340322fb8b9a1a6c
I20260926 20:49:08.017692 4523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:08.008470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35687 (local address 127.0.98.62:40229)
0926 20:49:08.008518 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:08.008520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:08.008774 (+ 254us) server_negotiation.cc:407] Connection header received
0926 20:49:08.009306 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:08.009308 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:08.009321 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:08.009377 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:08.010360 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:08.010879 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:08.011980 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:08.012175 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:08.012270 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:08.014447 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:08.014448 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:08.014449 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:08.014458 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:08.016117 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:08.016619 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:08.016620 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:08.016620 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:08.016670 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:08.016961 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:08.016961 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:08.016962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:08.017057 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:08.017129 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:08.017474 (+ 345us) server_negotiation.cc:299] Negotiation successful
0926 20:49:08.017572 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 20:49:08.017714 4809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:08.008457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40229 (local address 127.0.98.2:35687)
0926 20:49:08.008637 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:08.008672 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:08.009193 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:08.009447 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:08.009451 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:08.009746 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:08.010242 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:08.010247 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:08.010979 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:08.010980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:08.011891 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:08.011893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:08.012281 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:08.012282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:08.012415 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:08.012889 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:08.012897 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:08.014344 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:08.016196 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:08.016203 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:08.016206 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:08.016542 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:08.016737 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:08.016738 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:08.016738 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:08.016890 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:08.017120 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:08.017124 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:08.017318 (+ 194us) client_negotiation.cc:769] Sending connection context
0926 20:49:08.017431 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 20:49:08.017553 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":84,"threads_started":1}
I20260926 20:49:08.018235 4807 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40229
I20260926 20:49:08.018342 4807 heartbeater.cc:464] Registering TS with master...
I20260926 20:49:08.018498 4807 heartbeater.cc:511] Master 127.0.98.62:40229 requested a full tablet report, sending...
I20260926 20:49:08.018844 4484 ts_manager.cc:194] Registered new tserver with Master: 121b1093d17048f7a64b641f4ec5e9dd (127.0.98.2:37797)
I20260926 20:49:08.019209 4484 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35687
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 20:49:08.040503 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40229
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:49:08.121027 4812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:08.121230 4812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:08.121253 4812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:08.122426 4812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:08.122469 4812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:08.122496 4812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:49:08.124019 4812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:08.124254 4812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:08.124497 4812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:08.129480 4817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:08.129438 4820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:08.129428 4818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:08.129928 4812 server_base.cc:1034] running on GCE node
I20260926 20:49:08.130141 4812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:08.130327 4812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:08.131522 4812 hybrid_clock.cc:648] HybridClock initialized: now 1790455748131503 us; error 29 us; skew 500 ppm
Sep 26 20:49:08 dist-test-slave-hwlj krb5kdc[4440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455748, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:49:08.134096 4812 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 20:49:08.134490 4812 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:08.134562 4812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:08.134717 4812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:49:08.135844 4812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/ts-2/data/instance:
uuid: "be1c44796ebd456697f8ddec7620c54d"
format_stamp: "Formatted at 2026-09-26 20:49:08 on dist-test-slave-hwlj"
server_key: "83c31aada67022e4eec3c0ce4b9eb54a"
server_key_iv: "9528081742ef172d9ddf56c433a07f5a"
server_key_version: "encryptionkey@0"
I20260926 20:49:08.136240 4812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/ts-2/wal/instance:
uuid: "be1c44796ebd456697f8ddec7620c54d"
format_stamp: "Formatted at 2026-09-26 20:49:08 on dist-test-slave-hwlj"
server_key: "83c31aada67022e4eec3c0ce4b9eb54a"
server_key_iv: "9528081742ef172d9ddf56c433a07f5a"
server_key_version: "encryptionkey@0"
I20260926 20:49:08.137511 4812 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 20:49:08.138201 4826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:08.138384 4812 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 20:49:08.138484 4812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/ts-2/wal
uuid: "be1c44796ebd456697f8ddec7620c54d"
format_stamp: "Formatted at 2026-09-26 20:49:08 on dist-test-slave-hwlj"
server_key: "83c31aada67022e4eec3c0ce4b9eb54a"
server_key_iv: "9528081742ef172d9ddf56c433a07f5a"
server_key_version: "encryptionkey@0"
I20260926 20:49:08.138559 4812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:08.152500 4812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:08.154234 4812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:08.154399 4812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:08.154625 4812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:49:08.154927 4812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:49:08.154979 4812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:08.155020 4812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:49:08.155049 4812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:08.160844 4812 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37535
I20260926 20:49:08.160871 4939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37535 every 8 connection(s)
I20260926 20:49:08.161139 4812 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
I20260926 20:49:08.165344 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 4812
I20260926 20:49:08.165457 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/ts-2/wal/instance
I20260926 20:49:08.165755 392 external_mini_cluster.cc:1428] Setting key a9e930878c5a08cec4e9eae461b49f60
Sep 26 20:49:08 dist-test-slave-hwlj krb5kdc[4440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455748, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:49:08.170231 4523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:08.161822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47027 (local address 127.0.98.62:40229)
0926 20:49:08.161884 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:08.161887 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:08.162045 (+ 158us) server_negotiation.cc:407] Connection header received
0926 20:49:08.162520 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:08.162520 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:08.162531 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:08.162580 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:08.163511 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:08.164067 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:08.165213 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:08.165455 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:08.165537 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:08.167648 (+ 2111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:08.167650 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:08.167650 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:08.167657 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:08.168884 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:08.169294 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:08.169295 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:08.169295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:08.169356 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:08.169610 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:08.169610 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:08.169611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:08.169724 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:08.169793 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:08.170066 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 20:49:08.170141 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 20:49:08.170413 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:08.161782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40229 (local address 127.0.98.3:47027)
0926 20:49:08.161933 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:08.161972 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:08.162440 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:08.162639 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:08.162642 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:08.162890 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:08.163436 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:08.163442 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:08.164142 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:08.164142 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:08.165050 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:08.165053 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:08.165534 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:08.165535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:08.165665 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:08.166198 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:08.166209 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:08.167555 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:08.168989 (+ 1434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:08.168996 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:08.168998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:08.169205 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:08.169434 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:08.169435 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:08.169435 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:08.169538 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:08.169799 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:08.169804 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:08.169938 (+ 134us) client_negotiation.cc:769] Sending connection context
0926 20:49:08.170046 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 20:49:08.170195 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":84,"threads_started":1}
I20260926 20:49:08.170861 4940 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40229
I20260926 20:49:08.170949 4940 heartbeater.cc:464] Registering TS with master...
I20260926 20:49:08.171058 4940 heartbeater.cc:511] Master 127.0.98.62:40229 requested a full tablet report, sending...
I20260926 20:49:08.171375 4484 ts_manager.cc:194] Registered new tserver with Master: be1c44796ebd456697f8ddec7620c54d (127.0.98.3:37535)
I20260926 20:49:08.171693 4484 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47027
I20260926 20:49:08.177500 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:49:08.184772 4523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:08.178772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40390 (local address 127.0.98.62:40229)
0926 20:49:08.178810 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:08.178811 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:08.178967 (+ 156us) server_negotiation.cc:407] Connection header received
0926 20:49:08.179035 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:08.179036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:08.179046 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:08.179116 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:08.179665 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:08.180273 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:08.181041 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:08.181232 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:08.181301 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:08.182412 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:08.182414 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:08.182414 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:08.182421 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:08.183559 (+ 1138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:08.183986 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:08.183987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:08.183988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:08.184046 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:08.184241 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:08.184242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:08.184243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:08.184387 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:08.184500 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:08.184596 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 20:49:08.184671 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 20:49:08.186527 4484 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40390:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:49:08.188082 4484 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:49:08.194473 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:08.190488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53850 (local address 127.0.98.3:37535)
0926 20:49:08.190846 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:08.190851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:08.190861 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:49:08.190900 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:08.190902 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:08.190918 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:08.191003 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:49:08.191101 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:08.192044 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:08.193489 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:08.194084 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:08.194170 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:08.194230 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:08.194346 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 20:49:08.194384 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":70,"threads_started":1}
I20260926 20:49:08.194469 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:08.190238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37535 (local address 127.0.0.1:53850)
0926 20:49:08.190685 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:08.190694 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:08.190734 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:08.190999 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:08.191000 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:08.191002 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:49:08.191043 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:08.191044 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:08.192133 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:08.192134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:08.193391 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:08.193393 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:08.194170 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:08.194171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:08.194246 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:08.194259 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 20:49:08.194332 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 20:49:08.194385 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"mutex_wait_us":33,"thread_start_us":56,"threads_started":1}
I20260926 20:49:08.194681 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:08.190233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43047 (local address 127.0.0.1:34186)
0926 20:49:08.190477 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:08.190519 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:08.190577 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:08.190786 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:08.190789 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:08.190792 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:49:08.190874 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:08.190876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:08.192377 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:08.192378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:08.193388 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:08.193390 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:08.194423 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:08.194423 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:08.194489 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:08.194494 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:49:08.194541 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 20:49:08.194587 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":57,"threads_started":1}
I20260926 20:49:08.194922 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:08.190341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34186 (local address 127.0.98.1:43047)
0926 20:49:08.190546 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:08.190551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:08.190576 (+ 25us) server_negotiation.cc:407] Connection header received
0926 20:49:08.190667 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:08.190670 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:08.190686 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:08.190778 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:49:08.190949 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:08.192265 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:08.193472 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:08.194238 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:08.194323 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:08.194383 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:08.194785 (+ 402us) server_negotiation.cc:299] Negotiation successful
0926 20:49:08.194826 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":125,"threads_started":1}
I20260926 20:49:08.195250 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:08.190234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37797 (local address 127.0.0.1:59630)
0926 20:49:08.190553 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:08.190565 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:08.190618 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:08.191100 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:08.191101 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:08.191104 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:49:08.191153 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:08.191154 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:08.192449 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:08.192450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:08.193518 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:08.193520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:08.194420 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:08.194422 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:08.194489 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:08.194493 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:49:08.195141 (+ 648us) client_negotiation.cc:240] Negotiation successful
0926 20:49:08.195187 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":3,"thread_start_us":67,"threads_started":1}
I20260926 20:49:08.195209 4874 tablet_service.cc:1467] Processing CreateTablet for tablet 31f52862a09b418db0d2be33fd39c339 (DEFAULT_TABLE table=test-table [id=4de1a0b10d6f449ca5989ad427040ed3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:49:08.195555 4874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31f52862a09b418db0d2be33fd39c339. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:49:08.195515 4608 tablet_service.cc:1467] Processing CreateTablet for tablet 31f52862a09b418db0d2be33fd39c339 (DEFAULT_TABLE table=test-table [id=4de1a0b10d6f449ca5989ad427040ed3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:49:08.195648 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:08.190666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59630 (local address 127.0.98.2:37797)
0926 20:49:08.190868 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:08.190872 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:08.190880 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:49:08.190916 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:08.190920 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:08.190935 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:08.191013 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:49:08.191231 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:08.192349 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:08.193596 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:08.194287 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:08.194346 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:08.194392 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:08.195162 (+ 770us) server_negotiation.cc:299] Negotiation successful
0926 20:49:08.195204 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":70,"threads_started":1}
I20260926 20:49:08.195845 4608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31f52862a09b418db0d2be33fd39c339. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:49:08.196249 4741 tablet_service.cc:1467] Processing CreateTablet for tablet 31f52862a09b418db0d2be33fd39c339 (DEFAULT_TABLE table=test-table [id=4de1a0b10d6f449ca5989ad427040ed3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:49:08.196568 4741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31f52862a09b418db0d2be33fd39c339. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:49:08.198911 4955 tablet_bootstrap.cc:492] T 31f52862a09b418db0d2be33fd39c339 P f0c9456ff6084ab98db1d8c656ba2ad9: Bootstrap starting.
I20260926 20:49:08.199124 4954 tablet_bootstrap.cc:492] T 31f52862a09b418db0d2be33fd39c339 P be1c44796ebd456697f8ddec7620c54d: Bootstrap starting.
I20260926 20:49:08.199225 4956 tablet_bootstrap.cc:492] T 31f52862a09b418db0d2be33fd39c339 P 121b1093d17048f7a64b641f4ec5e9dd: Bootstrap starting.
I20260926 20:49:08.199712 4954 tablet_bootstrap.cc:654] T 31f52862a09b418db0d2be33fd39c339 P be1c44796ebd456697f8ddec7620c54d: Neither blocks nor log segments found. Creating new log.
I20260926 20:49:08.199798 4956 tablet_bootstrap.cc:654] T 31f52862a09b418db0d2be33fd39c339 P 121b1093d17048f7a64b641f4ec5e9dd: Neither blocks nor log segments found. Creating new log.
I20260926 20:49:08.200249 4954 log.cc:824] T 31f52862a09b418db0d2be33fd39c339 P be1c44796ebd456697f8ddec7620c54d: Log is configured to *not* fsync() on all Append() calls
I20260926 20:49:08.200358 4956 log.cc:824] T 31f52862a09b418db0d2be33fd39c339 P 121b1093d17048f7a64b641f4ec5e9dd: Log is configured to *not* fsync() on all Append() calls
I20260926 20:49:08.201066 4955 tablet_bootstrap.cc:654] T 31f52862a09b418db0d2be33fd39c339 P f0c9456ff6084ab98db1d8c656ba2ad9: Neither blocks nor log segments found. Creating new log.
I20260926 20:49:08.201076 4954 tablet_bootstrap.cc:492] T 31f52862a09b418db0d2be33fd39c339 P be1c44796ebd456697f8ddec7620c54d: No bootstrap required, opened a new log
I20260926 20:49:08.201136 4956 tablet_bootstrap.cc:492] T 31f52862a09b418db0d2be33fd39c339 P 121b1093d17048f7a64b641f4ec5e9dd: No bootstrap required, opened a new log
I20260926 20:49:08.201148 4954 ts_tablet_manager.cc:1397] T 31f52862a09b418db0d2be33fd39c339 P be1c44796ebd456697f8ddec7620c54d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 20:49:08.201196 4956 ts_tablet_manager.cc:1397] T 31f52862a09b418db0d2be33fd39c339 P 121b1093d17048f7a64b641f4ec5e9dd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:49:08.201653 4955 log.cc:824] T 31f52862a09b418db0d2be33fd39c339 P f0c9456ff6084ab98db1d8c656ba2ad9: Log is configured to *not* fsync() on all Append() calls
I20260926 20:49:08.203104 4954 raft_consensus.cc:348] T 31f52862a09b418db0d2be33fd39c339 P be1c44796ebd456697f8ddec7620c54d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1c44796ebd456697f8ddec7620c54d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37535 } } peers { permanent_uuid: "121b1093d17048f7a64b641f4ec5e9dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37797 } } peers { permanent_uuid: "f0c9456ff6084ab98db1d8c656ba2ad9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43047 } }
I20260926 20:49:08.203104 4956 raft_consensus.cc:348] T 31f52862a09b418db0d2be33fd39c339 P 121b1093d17048f7a64b641f4ec5e9dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1c44796ebd456697f8ddec7620c54d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37535 } } peers { permanent_uuid: "121b1093d17048f7a64b641f4ec5e9dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37797 } } peers { permanent_uuid: "f0c9456ff6084ab98db1d8c656ba2ad9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43047 } }
I20260926 20:49:08.203275 4954 raft_consensus.cc:374] T 31f52862a09b418db0d2be33fd39c339 P be1c44796ebd456697f8ddec7620c54d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:49:08.203275 4956 raft_consensus.cc:374] T 31f52862a09b418db0d2be33fd39c339 P 121b1093d17048f7a64b641f4ec5e9dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:49:08.203310 4956 raft_consensus.cc:729] T 31f52862a09b418db0d2be33fd39c339 P 121b1093d17048f7a64b641f4ec5e9dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 121b1093d17048f7a64b641f4ec5e9dd, State: Initialized, Role: FOLLOWER
I20260926 20:49:08.203311 4954 raft_consensus.cc:729] T 31f52862a09b418db0d2be33fd39c339 P be1c44796ebd456697f8ddec7620c54d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be1c44796ebd456697f8ddec7620c54d, State: Initialized, Role: FOLLOWER
I20260926 20:49:08.203461 4956 consensus_queue.cc:260] T 31f52862a09b418db0d2be33fd39c339 P 121b1093d17048f7a64b641f4ec5e9dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1c44796ebd456697f8ddec7620c54d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37535 } } peers { permanent_uuid: "121b1093d17048f7a64b641f4ec5e9dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37797 } } peers { permanent_uuid: "f0c9456ff6084ab98db1d8c656ba2ad9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43047 } }
I20260926 20:49:08.203449 4954 consensus_queue.cc:260] T 31f52862a09b418db0d2be33fd39c339 P be1c44796ebd456697f8ddec7620c54d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1c44796ebd456697f8ddec7620c54d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37535 } } peers { permanent_uuid: "121b1093d17048f7a64b641f4ec5e9dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37797 } } peers { permanent_uuid: "f0c9456ff6084ab98db1d8c656ba2ad9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43047 } }
I20260926 20:49:08.203735 4954 ts_tablet_manager.cc:1428] T 31f52862a09b418db0d2be33fd39c339 P be1c44796ebd456697f8ddec7620c54d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 20:49:08.203735 4956 ts_tablet_manager.cc:1428] T 31f52862a09b418db0d2be33fd39c339 P 121b1093d17048f7a64b641f4ec5e9dd: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 20:49:08.203825 4807 heartbeater.cc:503] Master 127.0.98.62:40229 was elected leader, sending a full tablet report...
I20260926 20:49:08.203879 4940 heartbeater.cc:503] Master 127.0.98.62:40229 was elected leader, sending a full tablet report...
I20260926 20:49:08.204454 4955 tablet_bootstrap.cc:492] T 31f52862a09b418db0d2be33fd39c339 P f0c9456ff6084ab98db1d8c656ba2ad9: No bootstrap required, opened a new log
I20260926 20:49:08.204522 4955 ts_tablet_manager.cc:1397] T 31f52862a09b418db0d2be33fd39c339 P f0c9456ff6084ab98db1d8c656ba2ad9: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260926 20:49:08.206490 4955 raft_consensus.cc:348] T 31f52862a09b418db0d2be33fd39c339 P f0c9456ff6084ab98db1d8c656ba2ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1c44796ebd456697f8ddec7620c54d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37535 } } peers { permanent_uuid: "121b1093d17048f7a64b641f4ec5e9dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37797 } } peers { permanent_uuid: "f0c9456ff6084ab98db1d8c656ba2ad9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43047 } }
I20260926 20:49:08.206646 4955 raft_consensus.cc:374] T 31f52862a09b418db0d2be33fd39c339 P f0c9456ff6084ab98db1d8c656ba2ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:49:08.206692 4955 raft_consensus.cc:729] T 31f52862a09b418db0d2be33fd39c339 P f0c9456ff6084ab98db1d8c656ba2ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0c9456ff6084ab98db1d8c656ba2ad9, State: Initialized, Role: FOLLOWER
I20260926 20:49:08.206808 4955 consensus_queue.cc:260] T 31f52862a09b418db0d2be33fd39c339 P f0c9456ff6084ab98db1d8c656ba2ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1c44796ebd456697f8ddec7620c54d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37535 } } peers { permanent_uuid: "121b1093d17048f7a64b641f4ec5e9dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37797 } } peers { permanent_uuid: "f0c9456ff6084ab98db1d8c656ba2ad9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43047 } }
I20260926 20:49:08.207084 4674 heartbeater.cc:503] Master 127.0.98.62:40229 was elected leader, sending a full tablet report...
I20260926 20:49:08.207089 4955 ts_tablet_manager.cc:1428] T 31f52862a09b418db0d2be33fd39c339 P f0c9456ff6084ab98db1d8c656ba2ad9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260926 20:49:08.258299 4808 tablet.cc:2369] T 31f52862a09b418db0d2be33fd39c339 P 121b1093d17048f7a64b641f4ec5e9dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:49:08.352293 4675 tablet.cc:2369] T 31f52862a09b418db0d2be33fd39c339 P f0c9456ff6084ab98db1d8c656ba2ad9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:49:08.411706 4941 tablet.cc:2369] T 31f52862a09b418db0d2be33fd39c339 P be1c44796ebd456697f8ddec7620c54d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:49:08.540516 4961 raft_consensus.cc:482] T 31f52862a09b418db0d2be33fd39c339 P be1c44796ebd456697f8ddec7620c54d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:49:08.540639 4961 raft_consensus.cc:504] T 31f52862a09b418db0d2be33fd39c339 P be1c44796ebd456697f8ddec7620c54d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1c44796ebd456697f8ddec7620c54d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37535 } } peers { permanent_uuid: "121b1093d17048f7a64b641f4ec5e9dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37797 } } peers { permanent_uuid: "f0c9456ff6084ab98db1d8c656ba2ad9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43047 } }
I20260926 20:49:08.541071 4961 leader_election.cc:288] T 31f52862a09b418db0d2be33fd39c339 P be1c44796ebd456697f8ddec7620c54d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 121b1093d17048f7a64b641f4ec5e9dd (127.0.98.2:37797), f0c9456ff6084ab98db1d8c656ba2ad9 (127.0.98.1:43047)
I20260926 20:49:08.544797 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:08.541385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54959 (local address 127.0.98.2:37797)
0926 20:49:08.541435 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:08.541437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:08.541639 (+ 202us) server_negotiation.cc:407] Connection header received
0926 20:49:08.541669 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:08.541670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:08.541683 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:08.541736 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:49:08.541917 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:08.542489 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:08.543844 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:08.544398 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:08.544479 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:08.544489 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:08.544690 (+ 201us) server_negotiation.cc:299] Negotiation successful
0926 20:49:08.544726 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 20:49:08.544797 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:08.541389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45021 (local address 127.0.98.1:43047)
0926 20:49:08.541440 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:08.541442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:08.541491 (+ 49us) server_negotiation.cc:407] Connection header received
0926 20:49:08.541522 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:08.541523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:08.541538 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:08.541595 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:49:08.541825 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:08.542390 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:08.543845 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:08.544398 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:08.544479 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:08.544489 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:08.544690 (+ 201us) server_negotiation.cc:299] Negotiation successful
0926 20:49:08.544726 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 20:49:08.544852 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:08.541300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43047 (local address 127.0.98.3:45021)
0926 20:49:08.541377 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:08.541390 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:08.541434 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:08.541619 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:08.541620 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:08.541622 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:49:08.541697 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:08.541699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:08.542486 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:08.542487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:08.543724 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:08.543726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:08.544514 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:08.544515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:08.544604 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:08.544606 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 20:49:08.544683 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 20:49:08.544720 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":7}
I20260926 20:49:08.545061 4963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:08.541311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37797 (local address 127.0.98.3:54959)
0926 20:49:08.541525 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:08.541536 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:08.541595 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:08.541747 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:08.541747 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:08.541749 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:49:08.541829 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:08.541830 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:08.542600 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:08.542600 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:08.543724 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:08.543726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:08.544514 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:08.544515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:08.544600 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:08.544604 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:49:08.544675 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 20:49:08.544720 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":4,"thread_start_us":118,"threads_started":1}
I20260926 20:49:08.545193 4628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f52862a09b418db0d2be33fd39c339" candidate_uuid: "be1c44796ebd456697f8ddec7620c54d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0c9456ff6084ab98db1d8c656ba2ad9" is_pre_election: true
I20260926 20:49:08.545365 4628 raft_consensus.cc:2454] T 31f52862a09b418db0d2be33fd39c339 P f0c9456ff6084ab98db1d8c656ba2ad9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be1c44796ebd456697f8ddec7620c54d in term 0.
I20260926 20:49:08.545476 4761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f52862a09b418db0d2be33fd39c339" candidate_uuid: "be1c44796ebd456697f8ddec7620c54d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "121b1093d17048f7a64b641f4ec5e9dd" is_pre_election: true
I20260926 20:49:08.545629 4761 raft_consensus.cc:2454] T 31f52862a09b418db0d2be33fd39c339 P 121b1093d17048f7a64b641f4ec5e9dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be1c44796ebd456697f8ddec7620c54d in term 0.
I20260926 20:49:08.545627 4830 leader_election.cc:302] T 31f52862a09b418db0d2be33fd39c339 P be1c44796ebd456697f8ddec7620c54d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be1c44796ebd456697f8ddec7620c54d, f0c9456ff6084ab98db1d8c656ba2ad9; no voters:
I20260926 20:49:08.545761 4961 raft_consensus.cc:2781] T 31f52862a09b418db0d2be33fd39c339 P be1c44796ebd456697f8ddec7620c54d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:49:08.545823 4961 raft_consensus.cc:482] T 31f52862a09b418db0d2be33fd39c339 P be1c44796ebd456697f8ddec7620c54d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:49:08.545849 4961 raft_consensus.cc:3037] T 31f52862a09b418db0d2be33fd39c339 P be1c44796ebd456697f8ddec7620c54d [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:49:08.546525 4961 raft_consensus.cc:504] T 31f52862a09b418db0d2be33fd39c339 P be1c44796ebd456697f8ddec7620c54d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1c44796ebd456697f8ddec7620c54d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37535 } } peers { permanent_uuid: "121b1093d17048f7a64b641f4ec5e9dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37797 } } peers { permanent_uuid: "f0c9456ff6084ab98db1d8c656ba2ad9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43047 } }
I20260926 20:49:08.546664 4961 leader_election.cc:288] T 31f52862a09b418db0d2be33fd39c339 P be1c44796ebd456697f8ddec7620c54d [CANDIDATE]: Term 1 election: Requested vote from peers 121b1093d17048f7a64b641f4ec5e9dd (127.0.98.2:37797), f0c9456ff6084ab98db1d8c656ba2ad9 (127.0.98.1:43047)
I20260926 20:49:08.546789 4761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f52862a09b418db0d2be33fd39c339" candidate_uuid: "be1c44796ebd456697f8ddec7620c54d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "121b1093d17048f7a64b641f4ec5e9dd"
I20260926 20:49:08.546895 4628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f52862a09b418db0d2be33fd39c339" candidate_uuid: "be1c44796ebd456697f8ddec7620c54d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0c9456ff6084ab98db1d8c656ba2ad9"
I20260926 20:49:08.546883 4761 raft_consensus.cc:3037] T 31f52862a09b418db0d2be33fd39c339 P 121b1093d17048f7a64b641f4ec5e9dd [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:49:08.546964 4628 raft_consensus.cc:3037] T 31f52862a09b418db0d2be33fd39c339 P f0c9456ff6084ab98db1d8c656ba2ad9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:49:08.547792 4761 raft_consensus.cc:2454] T 31f52862a09b418db0d2be33fd39c339 P 121b1093d17048f7a64b641f4ec5e9dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be1c44796ebd456697f8ddec7620c54d in term 1.
I20260926 20:49:08.547856 4628 raft_consensus.cc:2454] T 31f52862a09b418db0d2be33fd39c339 P f0c9456ff6084ab98db1d8c656ba2ad9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be1c44796ebd456697f8ddec7620c54d in term 1.
I20260926 20:49:08.547976 4828 leader_election.cc:302] T 31f52862a09b418db0d2be33fd39c339 P be1c44796ebd456697f8ddec7620c54d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 121b1093d17048f7a64b641f4ec5e9dd, be1c44796ebd456697f8ddec7620c54d; no voters:
I20260926 20:49:08.548059 4961 raft_consensus.cc:2781] T 31f52862a09b418db0d2be33fd39c339 P be1c44796ebd456697f8ddec7620c54d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:49:08.548197 4961 raft_consensus.cc:686] T 31f52862a09b418db0d2be33fd39c339 P be1c44796ebd456697f8ddec7620c54d [term 1 LEADER]: Becoming Leader. State: Replica: be1c44796ebd456697f8ddec7620c54d, State: Running, Role: LEADER
I20260926 20:49:08.548272 4961 consensus_queue.cc:237] T 31f52862a09b418db0d2be33fd39c339 P be1c44796ebd456697f8ddec7620c54d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1c44796ebd456697f8ddec7620c54d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37535 } } peers { permanent_uuid: "121b1093d17048f7a64b641f4ec5e9dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37797 } } peers { permanent_uuid: "f0c9456ff6084ab98db1d8c656ba2ad9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43047 } }
I20260926 20:49:08.549150 4484 catalog_manager.cc:5495] T 31f52862a09b418db0d2be33fd39c339 P be1c44796ebd456697f8ddec7620c54d reported cstate change: term changed from 0 to 1, leader changed from <none> to be1c44796ebd456697f8ddec7620c54d (127.0.98.3). New cstate: current_term: 1 leader_uuid: "be1c44796ebd456697f8ddec7620c54d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1c44796ebd456697f8ddec7620c54d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "121b1093d17048f7a64b641f4ec5e9dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0c9456ff6084ab98db1d8c656ba2ad9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43047 } health_report { overall_health: UNKNOWN } } }
I20260926 20:49:08.634805 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:08.632280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53860 (local address 127.0.98.3:37535)
0926 20:49:08.632346 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:08.632348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:08.632356 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:49:08.632398 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:08.632399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:08.632422 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:08.632515 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:49:08.632670 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:08.633163 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:08.633945 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:08.634156 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:08.634230 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:08.634412 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:49:08.634532 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:49:08.634645 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:08.634662 (+ 17us) server_negotiation.cc:299] Negotiation successful
0926 20:49:08.634682 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 20:49:08.636687 4761 raft_consensus.cc:1261] T 31f52862a09b418db0d2be33fd39c339 P 121b1093d17048f7a64b641f4ec5e9dd [term 1 FOLLOWER]: Refusing update from remote peer be1c44796ebd456697f8ddec7620c54d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:49:08.636686 4628 raft_consensus.cc:1261] T 31f52862a09b418db0d2be33fd39c339 P f0c9456ff6084ab98db1d8c656ba2ad9 [term 1 FOLLOWER]: Refusing update from remote peer be1c44796ebd456697f8ddec7620c54d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:49:08.637091 4964 consensus_queue.cc:1035] T 31f52862a09b418db0d2be33fd39c339 P be1c44796ebd456697f8ddec7620c54d [LEADER]: Connected to new peer: Peer: permanent_uuid: "121b1093d17048f7a64b641f4ec5e9dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:49:08.637257 4961 consensus_queue.cc:1035] T 31f52862a09b418db0d2be33fd39c339 P be1c44796ebd456697f8ddec7620c54d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0c9456ff6084ab98db1d8c656ba2ad9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:49:08.640141 4969 mvcc.cc:205] Tried to move back new op lower bound from 7333706746413015040 to 7333706746054307840. Current Snapshot: MvccSnapshot[applied={T|T < 7333706746413015040 or (T in {7333706746413015040})}]
I20260926 20:49:08.642227 4483 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:49:08.642336 4483 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:49:08.643110 4483 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 75043a94e3474209977d45b6f4463bea: Sending DeleteTablet for 3 replicas of tablet 31f52862a09b418db0d2be33fd39c339
I20260926 20:49:08.643437 4608 tablet_service.cc:1514] Processing DeleteTablet for tablet 31f52862a09b418db0d2be33fd39c339 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:49:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34186
I20260926 20:49:08.643419 4874 tablet_service.cc:1514] Processing DeleteTablet for tablet 31f52862a09b418db0d2be33fd39c339 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:49:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53850
I20260926 20:49:08.643509 4741 tablet_service.cc:1514] Processing DeleteTablet for tablet 31f52862a09b418db0d2be33fd39c339 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:49:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59630
I20260926 20:49:08.643656 4978 tablet_replica.cc:330] T 31f52862a09b418db0d2be33fd39c339 P f0c9456ff6084ab98db1d8c656ba2ad9: stopping tablet replica
I20260926 20:49:08.643783 4978 raft_consensus.cc:2229] T 31f52862a09b418db0d2be33fd39c339 P f0c9456ff6084ab98db1d8c656ba2ad9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:49:08.643860 4980 tablet_replica.cc:330] T 31f52862a09b418db0d2be33fd39c339 P 121b1093d17048f7a64b641f4ec5e9dd: stopping tablet replica
I20260926 20:49:08.643908 4978 raft_consensus.cc:2258] T 31f52862a09b418db0d2be33fd39c339 P f0c9456ff6084ab98db1d8c656ba2ad9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:49:08.643939 4980 raft_consensus.cc:2229] T 31f52862a09b418db0d2be33fd39c339 P 121b1093d17048f7a64b641f4ec5e9dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:49:08.643980 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 4546
I20260926 20:49:08.644021 4980 raft_consensus.cc:2258] T 31f52862a09b418db0d2be33fd39c339 P 121b1093d17048f7a64b641f4ec5e9dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:49:08.645052 4980 ts_tablet_manager.cc:1905] T 31f52862a09b418db0d2be33fd39c339 P 121b1093d17048f7a64b641f4ec5e9dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:49:08.646517 4980 ts_tablet_manager.cc:1918] T 31f52862a09b418db0d2be33fd39c339 P 121b1093d17048f7a64b641f4ec5e9dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:49:08.646566 4980 log.cc:1196] T 31f52862a09b418db0d2be33fd39c339 P 121b1093d17048f7a64b641f4ec5e9dd: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455725523895-392-0/minicluster-data/ts-1/wal/wals/31f52862a09b418db0d2be33fd39c339
I20260926 20:49:08.646749 4980 ts_tablet_manager.cc:1939] T 31f52862a09b418db0d2be33fd39c339 P 121b1093d17048f7a64b641f4ec5e9dd: Deleting consensus metadata
I20260926 20:49:08.647189 4469 catalog_manager.cc:4841] TS 121b1093d17048f7a64b641f4ec5e9dd (127.0.98.2:37797): tablet 31f52862a09b418db0d2be33fd39c339 (table test-table [id=4de1a0b10d6f449ca5989ad427040ed3]) successfully deleted
I20260926 20:49:08.650230 4979 tablet_replica.cc:330] T 31f52862a09b418db0d2be33fd39c339 P be1c44796ebd456697f8ddec7620c54d: stopping tablet replica
I20260926 20:49:08.650364 4979 raft_consensus.cc:2229] T 31f52862a09b418db0d2be33fd39c339 P be1c44796ebd456697f8ddec7620c54d [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:49:08.650480 4979 raft_consensus.cc:2258] T 31f52862a09b418db0d2be33fd39c339 P be1c44796ebd456697f8ddec7620c54d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:49:08.650954 4979 ts_tablet_manager.cc:1905] T 31f52862a09b418db0d2be33fd39c339 P be1c44796ebd456697f8ddec7620c54d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 20:49:08.651916 4471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43047 (error 108)
I20260926 20:49:08.652014 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 4679
I20260926 20:49:08.653208 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:08.653076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43047 (local address 127.0.0.1:34194)
0926 20:49:08.653132 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:08.653171 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 20:49:08.653343 4471 catalog_manager.cc:4584] TS f0c9456ff6084ab98db1d8c656ba2ad9 (127.0.98.1:43047): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 31f52862a09b418db0d2be33fd39c339: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43047: connect: Connection refused (error 111)
I20260926 20:49:08.658515 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 4812
I20260926 20:49:08.658955 4979 ts_tablet_manager.cc:1918] T 31f52862a09b418db0d2be33fd39c339 P be1c44796ebd456697f8ddec7620c54d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:49:08.662709 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 4453
I20260926 20:49:08.662879 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:08.662719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37535 (local address 127.0.0.1:53872)
0926 20:49:08.662813 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:08.662844 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.3:37535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
2026-09-26T20:49:08Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2489 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T20:49:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:49:08Z Disabled control of system clock
I20260926 20:49:08.690411 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45291
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45291
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 20:49:08.770990 4985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:08.771214 4985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:08.771235 4985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:08.772341 4985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:49:08.772378 4985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:08.772390 4985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:08.772403 4985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:49:08.772415 4985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:49:08.773716 4985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45291
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:08.773913 4985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:08.774144 4985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:49:08.779172 4985 server_base.cc:1034] running on GCE node
W20260926 20:49:08.779143 4990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:08.779187 4991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:08.779143 4993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:08.779668 4985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:08.780031 4985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:08.781244 4985 hybrid_clock.cc:648] HybridClock initialized: now 1790455748781227 us; error 45 us; skew 500 ppm
I20260926 20:49:08.787897 4985 webserver.cc:460] Webserver started at http://127.0.98.62:39885/ using document root <none> and password file <none>
I20260926 20:49:08.788172 4985 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:08.788245 4985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:08.788412 4985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:49:08.789521 4985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/master-0/data/instance:
uuid: "2c64f1abc95b4ca5b18db6d789716b79"
format_stamp: "Formatted at 2026-09-26 20:49:08 on dist-test-slave-hwlj"
server_key: "380a5bcbf1820bace85fe6f490d949df"
server_key_iv: "5a880c50143672f6b40684124cf1d3be"
server_key_version: "encryptionkey@0"
I20260926 20:49:08.789917 4985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/master-0/wal/instance:
uuid: "2c64f1abc95b4ca5b18db6d789716b79"
format_stamp: "Formatted at 2026-09-26 20:49:08 on dist-test-slave-hwlj"
server_key: "380a5bcbf1820bace85fe6f490d949df"
server_key_iv: "5a880c50143672f6b40684124cf1d3be"
server_key_version: "encryptionkey@0"
I20260926 20:49:08.791239 4985 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:49:08.792124 4999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:08.792296 4985 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:08.792412 4985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "2c64f1abc95b4ca5b18db6d789716b79"
format_stamp: "Formatted at 2026-09-26 20:49:08 on dist-test-slave-hwlj"
server_key: "380a5bcbf1820bace85fe6f490d949df"
server_key_iv: "5a880c50143672f6b40684124cf1d3be"
server_key_version: "encryptionkey@0"
I20260926 20:49:08.792496 4985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:08.795913 4985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:08.796254 4985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:08.796423 4985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:08.800462 4985 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45291
I20260926 20:49:08.800491 5051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45291 every 8 connection(s)
I20260926 20:49:08.800830 4985 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:49:08.801486 5052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:49:08.804611 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 4985
I20260926 20:49:08.804709 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/master-0/wal/instance
I20260926 20:49:08.804893 392 external_mini_cluster.cc:1428] Setting key 122071e1dba82186c275ccdebaf363f5
I20260926 20:49:08.805006 5052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c64f1abc95b4ca5b18db6d789716b79: Bootstrap starting.
I20260926 20:49:08.805708 5052 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c64f1abc95b4ca5b18db6d789716b79: Neither blocks nor log segments found. Creating new log.
I20260926 20:49:08.805950 5052 log.cc:824] T 00000000000000000000000000000000 P 2c64f1abc95b4ca5b18db6d789716b79: Log is configured to *not* fsync() on all Append() calls
I20260926 20:49:08.806731 5052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c64f1abc95b4ca5b18db6d789716b79: No bootstrap required, opened a new log
I20260926 20:49:08.807574 5055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:08.805418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41214 (local address 127.0.98.62:45291)
0926 20:49:08.805602 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:08.805617 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:08.805641 (+ 24us) server_negotiation.cc:407] Connection header received
0926 20:49:08.806204 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:08.806213 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:08.806264 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:08.806425 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:08.807143 (+ 718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:08.807147 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:08.807147 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:49:08.807152 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:08.807199 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:49:08.807213 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:08.807307 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:08.807335 (+ 28us) server_negotiation.cc:299] Negotiation successful
0926 20:49:08.807389 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":87,"threads_started":1}
I20260926 20:49:08.808385 5052 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c64f1abc95b4ca5b18db6d789716b79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c64f1abc95b4ca5b18db6d789716b79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45291 } }
I20260926 20:49:08.808519 5052 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c64f1abc95b4ca5b18db6d789716b79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:49:08.808542 5052 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c64f1abc95b4ca5b18db6d789716b79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c64f1abc95b4ca5b18db6d789716b79, State: Initialized, Role: FOLLOWER
I20260926 20:49:08.808686 5052 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c64f1abc95b4ca5b18db6d789716b79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c64f1abc95b4ca5b18db6d789716b79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45291 } }
I20260926 20:49:08.808754 5052 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c64f1abc95b4ca5b18db6d789716b79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:49:08.808796 5052 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c64f1abc95b4ca5b18db6d789716b79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:49:08.808845 5052 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c64f1abc95b4ca5b18db6d789716b79 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:49:08.809640 5052 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c64f1abc95b4ca5b18db6d789716b79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c64f1abc95b4ca5b18db6d789716b79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45291 } }
I20260926 20:49:08.809751 5052 leader_election.cc:302] T 00000000000000000000000000000000 P 2c64f1abc95b4ca5b18db6d789716b79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c64f1abc95b4ca5b18db6d789716b79; no voters:
I20260926 20:49:08.809902 5052 leader_election.cc:288] T 00000000000000000000000000000000 P 2c64f1abc95b4ca5b18db6d789716b79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:49:08.809966 5057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c64f1abc95b4ca5b18db6d789716b79 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:49:08.810101 5057 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c64f1abc95b4ca5b18db6d789716b79 [term 1 LEADER]: Becoming Leader. State: Replica: 2c64f1abc95b4ca5b18db6d789716b79, State: Running, Role: LEADER
I20260926 20:49:08.810156 5052 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c64f1abc95b4ca5b18db6d789716b79 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:49:08.810189 5057 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c64f1abc95b4ca5b18db6d789716b79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c64f1abc95b4ca5b18db6d789716b79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45291 } }
I20260926 20:49:08.810559 5058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c64f1abc95b4ca5b18db6d789716b79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c64f1abc95b4ca5b18db6d789716b79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c64f1abc95b4ca5b18db6d789716b79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45291 } } }
I20260926 20:49:08.810560 5059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c64f1abc95b4ca5b18db6d789716b79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c64f1abc95b4ca5b18db6d789716b79. Latest consensus state: current_term: 1 leader_uuid: "2c64f1abc95b4ca5b18db6d789716b79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c64f1abc95b4ca5b18db6d789716b79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45291 } } }
I20260926 20:49:08.810678 5058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c64f1abc95b4ca5b18db6d789716b79 [sys.catalog]: This master's current role is: LEADER
I20260926 20:49:08.810684 5059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c64f1abc95b4ca5b18db6d789716b79 [sys.catalog]: This master's current role is: LEADER
I20260926 20:49:08.810899 5065 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:49:08.811333 5065 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:49:08.812834 5065 catalog_manager.cc:1347] Generated new cluster ID: abb98a7cbba647fd90387269bbedf64e
I20260926 20:49:08.812902 5065 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:49:08.827967 5065 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:49:08.828169 5065 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:49:08.842795 5065 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c64f1abc95b4ca5b18db6d789716b79: Generated new TSK 0
I20260926 20:49:08.843026 5065 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:49:08.851823 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45291
--builtin_ntp_servers=127.0.98.20:37725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 20:49:08.933820 5076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:08.934034 5076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:08.934056 5076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:08.935251 5076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:08.935292 5076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:08.935319 5076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:49:08.936771 5076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:08.937017 5076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:08.937330 5076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:08.942510 5082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:08.942517 5084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:08.942539 5076 server_base.cc:1034] running on GCE node
W20260926 20:49:08.942528 5081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:08.943045 5076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:08.943269 5076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:08.944454 5076 hybrid_clock.cc:648] HybridClock initialized: now 1790455748944435 us; error 29 us; skew 500 ppm
I20260926 20:49:08.945909 5076 webserver.cc:460] Webserver started at http://127.0.98.1:37037/ using document root <none> and password file <none>
I20260926 20:49:08.946156 5076 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:08.946225 5076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:08.946391 5076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:49:08.947480 5076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/ts-0/data/instance:
uuid: "027d6b7f71504dd28306d2962d794785"
format_stamp: "Formatted at 2026-09-26 20:49:08 on dist-test-slave-hwlj"
server_key: "c520dfa6626351cd16a94f5378334859"
server_key_iv: "7922d5da41c41eebb6c3d8f2cff7334d"
server_key_version: "encryptionkey@0"
I20260926 20:49:08.947912 5076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/ts-0/wal/instance:
uuid: "027d6b7f71504dd28306d2962d794785"
format_stamp: "Formatted at 2026-09-26 20:49:08 on dist-test-slave-hwlj"
server_key: "c520dfa6626351cd16a94f5378334859"
server_key_iv: "7922d5da41c41eebb6c3d8f2cff7334d"
server_key_version: "encryptionkey@0"
I20260926 20:49:08.949275 5076 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:49:08.949982 5090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:08.950181 5076 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 20:49:08.950304 5076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/ts-0/wal
uuid: "027d6b7f71504dd28306d2962d794785"
format_stamp: "Formatted at 2026-09-26 20:49:08 on dist-test-slave-hwlj"
server_key: "c520dfa6626351cd16a94f5378334859"
server_key_iv: "7922d5da41c41eebb6c3d8f2cff7334d"
server_key_version: "encryptionkey@0"
I20260926 20:49:08.950395 5076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:08.953820 5076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:08.954085 5076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:08.954231 5076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:08.954437 5076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:49:08.954746 5076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:49:08.954797 5076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:08.954836 5076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:49:08.954864 5076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:08.960575 5076 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38499
I20260926 20:49:08.960611 5203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38499 every 8 connection(s)
I20260926 20:49:08.960955 5076 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:49:08.963989 5055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:08.961662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42853 (local address 127.0.98.62:45291)
0926 20:49:08.961710 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:08.961713 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:08.961978 (+ 265us) server_negotiation.cc:407] Connection header received
0926 20:49:08.962428 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:08.962429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:08.962435 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:08.962488 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:08.963700 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:08.963701 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:08.963702 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:49:08.963706 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:08.963720 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:49:08.963724 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:08.963837 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:08.963904 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 20:49:08.963917 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 20:49:08.964184 5206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:08.961662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45291 (local address 127.0.98.1:42853)
0926 20:49:08.961849 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:08.961882 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:08.962347 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:08.962539 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:08.962542 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:08.962550 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:08.963552 (+ 1002us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:49:08.963553 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:08.963580 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:49:08.963609 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:08.963836 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:08.963839 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:08.963842 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:49:08.963900 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 20:49:08.963950 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":86,"threads_started":1}
I20260926 20:49:08.964789 5204 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45291
I20260926 20:49:08.964896 5204 heartbeater.cc:464] Registering TS with master...
I20260926 20:49:08.965044 5204 heartbeater.cc:511] Master 127.0.98.62:45291 requested a full tablet report, sending...
I20260926 20:49:08.965570 5016 ts_manager.cc:194] Registered new tserver with Master: 027d6b7f71504dd28306d2962d794785 (127.0.98.1:38499)
I20260926 20:49:08.966610 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 5076
I20260926 20:49:08.966696 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/ts-0/wal/instance
I20260926 20:49:08.966867 392 external_mini_cluster.cc:1428] Setting key ef0af58c48497be73c83657952196273
I20260926 20:49:08.967697 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45291
--builtin_ntp_servers=127.0.98.20:37725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 20:49:09.046766 5207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:09.047003 5207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:09.047044 5207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:09.048308 5207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:09.048372 5207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:09.048424 5207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:49:09.049809 5207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:09.050078 5207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:09.050365 5207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:09.055567 5213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:09.055614 5212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:09.055591 5215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:09.055780 5207 server_base.cc:1034] running on GCE node
I20260926 20:49:09.055986 5207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:09.056226 5207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:09.057363 5207 hybrid_clock.cc:648] HybridClock initialized: now 1790455749057343 us; error 37 us; skew 500 ppm
I20260926 20:49:09.058512 5207 webserver.cc:460] Webserver started at http://127.0.98.2:42943/ using document root <none> and password file <none>
I20260926 20:49:09.058756 5207 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:09.058799 5207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:09.058952 5207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:49:09.060081 5207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/ts-1/data/instance:
uuid: "93b6a47508be47669c5f292bb70a5ea8"
format_stamp: "Formatted at 2026-09-26 20:49:09 on dist-test-slave-hwlj"
server_key: "40a47faf2ac797bb527e1f9c159d10b4"
server_key_iv: "8b163dd5e1a027ca33116adf9706f81b"
server_key_version: "encryptionkey@0"
I20260926 20:49:09.060436 5207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/ts-1/wal/instance:
uuid: "93b6a47508be47669c5f292bb70a5ea8"
format_stamp: "Formatted at 2026-09-26 20:49:09 on dist-test-slave-hwlj"
server_key: "40a47faf2ac797bb527e1f9c159d10b4"
server_key_iv: "8b163dd5e1a027ca33116adf9706f81b"
server_key_version: "encryptionkey@0"
I20260926 20:49:09.061741 5207 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:49:09.062441 5221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:09.062626 5207 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:49:09.062695 5207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/ts-1/wal
uuid: "93b6a47508be47669c5f292bb70a5ea8"
format_stamp: "Formatted at 2026-09-26 20:49:09 on dist-test-slave-hwlj"
server_key: "40a47faf2ac797bb527e1f9c159d10b4"
server_key_iv: "8b163dd5e1a027ca33116adf9706f81b"
server_key_version: "encryptionkey@0"
I20260926 20:49:09.062777 5207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:09.066016 5207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:09.066262 5207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:09.066397 5207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:09.066612 5207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:49:09.066906 5207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:49:09.066937 5207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:09.066991 5207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:49:09.067020 5207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:09.072665 5207 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35073
I20260926 20:49:09.072708 5334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35073 every 8 connection(s)
I20260926 20:49:09.073002 5207 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
I20260926 20:49:09.075876 5055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:09.073678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42123 (local address 127.0.98.62:45291)
0926 20:49:09.073737 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:09.073739 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:09.073975 (+ 236us) server_negotiation.cc:407] Connection header received
0926 20:49:09.074407 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:09.074408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:09.074414 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:09.074472 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:09.075624 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:09.075624 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:09.075625 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:49:09.075627 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:09.075637 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:49:09.075640 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:09.075675 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:09.075797 (+ 122us) server_negotiation.cc:299] Negotiation successful
0926 20:49:09.075812 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 20:49:09.076061 5337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:09.073629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45291 (local address 127.0.98.2:42123)
0926 20:49:09.073844 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:09.073883 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:09.074322 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:09.074536 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:09.074539 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:09.074547 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:09.075498 (+ 951us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:49:09.075499 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:09.075526 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:49:09.075552 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:09.075683 (+ 131us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:09.075686 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:09.075688 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 20:49:09.075805 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 20:49:09.075865 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":130,"threads_started":1}
I20260926 20:49:09.076555 5335 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45291
I20260926 20:49:09.076646 5335 heartbeater.cc:464] Registering TS with master...
I20260926 20:49:09.076792 5335 heartbeater.cc:511] Master 127.0.98.62:45291 requested a full tablet report, sending...
I20260926 20:49:09.077152 5016 ts_manager.cc:194] Registered new tserver with Master: 93b6a47508be47669c5f292bb70a5ea8 (127.0.98.2:35073)
I20260926 20:49:09.082002 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 5207
I20260926 20:49:09.082121 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/ts-1/wal/instance
I20260926 20:49:09.082309 392 external_mini_cluster.cc:1428] Setting key 6a8e558500edbd91785435b63fb73a9e
I20260926 20:49:09.083308 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45291
--builtin_ntp_servers=127.0.98.20:37725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 20:49:09.165532 5338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:09.165745 5338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:09.165768 5338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:09.166952 5338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:09.166989 5338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:09.167047 5338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:49:09.168454 5338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:09.168695 5338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:09.168977 5338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:09.173957 5343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:09.174011 5338 server_base.cc:1034] running on GCE node
W20260926 20:49:09.174091 5346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:09.173966 5344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:09.174378 5338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:09.174657 5338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:09.175861 5338 hybrid_clock.cc:648] HybridClock initialized: now 1790455749175841 us; error 29 us; skew 500 ppm
I20260926 20:49:09.177155 5338 webserver.cc:460] Webserver started at http://127.0.98.3:34547/ using document root <none> and password file <none>
I20260926 20:49:09.177407 5338 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:09.177476 5338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:09.177621 5338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:49:09.178660 5338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/ts-2/data/instance:
uuid: "562ae2267a4c4f789c46f00bfe367e4a"
format_stamp: "Formatted at 2026-09-26 20:49:09 on dist-test-slave-hwlj"
server_key: "c3951c81fc838ab54584d12128f26f15"
server_key_iv: "4826b9aef97b568c6a26283895fc52d0"
server_key_version: "encryptionkey@0"
I20260926 20:49:09.179061 5338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/ts-2/wal/instance:
uuid: "562ae2267a4c4f789c46f00bfe367e4a"
format_stamp: "Formatted at 2026-09-26 20:49:09 on dist-test-slave-hwlj"
server_key: "c3951c81fc838ab54584d12128f26f15"
server_key_iv: "4826b9aef97b568c6a26283895fc52d0"
server_key_version: "encryptionkey@0"
I20260926 20:49:09.180409 5338 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:49:09.181123 5352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:09.181293 5338 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:49:09.181389 5338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/ts-2/wal
uuid: "562ae2267a4c4f789c46f00bfe367e4a"
format_stamp: "Formatted at 2026-09-26 20:49:09 on dist-test-slave-hwlj"
server_key: "c3951c81fc838ab54584d12128f26f15"
server_key_iv: "4826b9aef97b568c6a26283895fc52d0"
server_key_version: "encryptionkey@0"
I20260926 20:49:09.181458 5338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:09.184986 5338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:09.185257 5338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:09.185395 5338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:09.185582 5338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:49:09.185896 5338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:49:09.185950 5338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:09.185990 5338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:49:09.186018 5338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:09.191838 5338 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36307
I20260926 20:49:09.191900 5465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36307 every 8 connection(s)
I20260926 20:49:09.192232 5338 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
I20260926 20:49:09.195212 5055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:09.192938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57939 (local address 127.0.98.62:45291)
0926 20:49:09.193003 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:09.193006 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:09.193284 (+ 278us) server_negotiation.cc:407] Connection header received
0926 20:49:09.193725 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:09.193726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:09.193732 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:09.193777 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:09.195017 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:09.195017 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:09.195018 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:49:09.195021 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:09.195031 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:49:09.195034 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:09.195076 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:09.195150 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 20:49:09.195162 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 20:49:09.195423 5468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:09.192971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45291 (local address 127.0.98.3:57939)
0926 20:49:09.193165 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:09.193198 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:09.193645 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:09.193854 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:09.193857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:09.193865 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:09.194884 (+ 1019us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:49:09.194884 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:09.194912 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:49:09.194942 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:09.195083 (+ 141us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:09.195086 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:09.195089 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:49:09.195148 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 20:49:09.195196 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":102,"threads_started":1}
I20260926 20:49:09.195945 5466 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45291
I20260926 20:49:09.196033 5466 heartbeater.cc:464] Registering TS with master...
I20260926 20:49:09.196141 5466 heartbeater.cc:511] Master 127.0.98.62:45291 requested a full tablet report, sending...
I20260926 20:49:09.196481 5016 ts_manager.cc:194] Registered new tserver with Master: 562ae2267a4c4f789c46f00bfe367e4a (127.0.98.3:36307)
I20260926 20:49:09.197743 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 5338
I20260926 20:49:09.197819 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/ts-2/wal/instance
I20260926 20:49:09.198026 392 external_mini_cluster.cc:1428] Setting key e9bf36abd6a9a09f6faefb0b02d8453f
I20260926 20:49:09.198850 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:49:09.201205 5055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:09.200184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41230 (local address 127.0.98.62:45291)
0926 20:49:09.200235 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:09.200237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:09.200346 (+ 109us) server_negotiation.cc:407] Connection header received
0926 20:49:09.200410 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:09.200411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:09.200415 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:09.200459 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:09.201057 (+ 598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:09.201057 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:09.201058 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:49:09.201060 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:09.201070 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:49:09.201073 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:09.201110 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:09.201150 (+ 40us) server_negotiation.cc:299] Negotiation successful
0926 20:49:09.201160 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 20:49:09.202687 5016 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41230:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:49:09.205355 5016 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:49:09.209556 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:09.207801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38499 (local address 127.0.0.1:48708)
0926 20:49:09.207929 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:09.207972 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:09.208032 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:09.208599 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:09.208605 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:09.208612 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:09.209079 (+ 467us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:49:09.209079 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:09.209101 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:49:09.209140 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:09.209338 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:09.209344 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:09.209347 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:49:09.209473 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 20:49:09.209487 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":55,"threads_started":1}
I20260926 20:49:09.209595 5475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:09.207853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48708 (local address 127.0.98.1:38499)
0926 20:49:09.208149 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:09.208154 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:09.208166 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:49:09.208193 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:09.208196 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:09.208206 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:09.208281 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:09.209201 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:09.209208 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:09.209209 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:49:09.209213 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:09.209246 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:49:09.209274 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:09.209345 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:09.209457 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 20:49:09.209476 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":104,"threads_started":1}
I20260926 20:49:09.209781 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:09.208013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35073 (local address 127.0.0.1:51666)
0926 20:49:09.208417 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:09.208427 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:09.208471 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:09.208664 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:09.208664 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:09.208667 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:09.209397 (+ 730us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:49:09.209398 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:09.209404 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:49:09.209415 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:09.209668 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:09.209669 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:09.209670 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 20:49:09.209714 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 20:49:09.209727 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":66,"threads_started":1}
I20260926 20:49:09.210026 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:09.208103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51666 (local address 127.0.98.2:35073)
0926 20:49:09.208322 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:09.208326 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:09.208528 (+ 202us) server_negotiation.cc:407] Connection header received
0926 20:49:09.208557 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:09.208559 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:09.208570 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:09.208648 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:09.209522 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:09.209528 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:09.209528 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:49:09.209532 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:09.209564 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:49:09.209587 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:09.209644 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:09.209937 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 20:49:09.209952 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":84,"threads_started":1}
I20260926 20:49:09.210320 5138 tablet_service.cc:1467] Processing CreateTablet for tablet e4e9a3cec7d84d8896a97b1908d12240 (DEFAULT_TABLE table=test-table [id=1e3d38f0da844de6a7758dbdd8d15102]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:49:09.210711 5138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4e9a3cec7d84d8896a97b1908d12240. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:49:09.210678 5269 tablet_service.cc:1467] Processing CreateTablet for tablet e4e9a3cec7d84d8896a97b1908d12240 (DEFAULT_TABLE table=test-table [id=1e3d38f0da844de6a7758dbdd8d15102]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:49:09.210995 5269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4e9a3cec7d84d8896a97b1908d12240. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:49:09.213645 5481 tablet_bootstrap.cc:492] T e4e9a3cec7d84d8896a97b1908d12240 P 027d6b7f71504dd28306d2962d794785: Bootstrap starting.
I20260926 20:49:09.213644 5480 tablet_bootstrap.cc:492] T e4e9a3cec7d84d8896a97b1908d12240 P 93b6a47508be47669c5f292bb70a5ea8: Bootstrap starting.
I20260926 20:49:09.214344 5480 tablet_bootstrap.cc:654] T e4e9a3cec7d84d8896a97b1908d12240 P 93b6a47508be47669c5f292bb70a5ea8: Neither blocks nor log segments found. Creating new log.
I20260926 20:49:09.214375 5481 tablet_bootstrap.cc:654] T e4e9a3cec7d84d8896a97b1908d12240 P 027d6b7f71504dd28306d2962d794785: Neither blocks nor log segments found. Creating new log.
I20260926 20:49:09.214425 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:09.208170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36307 (local address 127.0.0.1:38402)
0926 20:49:09.208930 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:09.208938 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:09.209011 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:09.209435 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:09.209437 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:09.209439 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:09.213994 (+ 4555us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:49:09.213995 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:09.214004 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:49:09.214018 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:09.214283 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:09.214284 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:09.214285 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 20:49:09.214330 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 20:49:09.214344 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"thread_start_us":83,"threads_started":1}
I20260926 20:49:09.214668 5480 log.cc:824] T e4e9a3cec7d84d8896a97b1908d12240 P 93b6a47508be47669c5f292bb70a5ea8: Log is configured to *not* fsync() on all Append() calls
I20260926 20:49:09.214689 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:09.208283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38402 (local address 127.0.98.3:36307)
0926 20:49:09.208461 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:09.208465 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:09.209038 (+ 573us) server_negotiation.cc:407] Connection header received
0926 20:49:09.209069 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:09.209071 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:09.209081 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:09.209157 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:09.214121 (+ 4964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:09.214131 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:09.214132 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:49:09.214137 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:09.214174 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:49:09.214202 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:09.214257 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:09.214576 (+ 319us) server_negotiation.cc:299] Negotiation successful
0926 20:49:09.214598 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":78,"threads_started":1}
I20260926 20:49:09.214699 5481 log.cc:824] T e4e9a3cec7d84d8896a97b1908d12240 P 027d6b7f71504dd28306d2962d794785: Log is configured to *not* fsync() on all Append() calls
I20260926 20:49:09.215237 5400 tablet_service.cc:1467] Processing CreateTablet for tablet e4e9a3cec7d84d8896a97b1908d12240 (DEFAULT_TABLE table=test-table [id=1e3d38f0da844de6a7758dbdd8d15102]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:49:09.215471 5481 tablet_bootstrap.cc:492] T e4e9a3cec7d84d8896a97b1908d12240 P 027d6b7f71504dd28306d2962d794785: No bootstrap required, opened a new log
I20260926 20:49:09.215546 5400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4e9a3cec7d84d8896a97b1908d12240. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:49:09.215544 5481 ts_tablet_manager.cc:1397] T e4e9a3cec7d84d8896a97b1908d12240 P 027d6b7f71504dd28306d2962d794785: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:49:09.216137 5480 tablet_bootstrap.cc:492] T e4e9a3cec7d84d8896a97b1908d12240 P 93b6a47508be47669c5f292bb70a5ea8: No bootstrap required, opened a new log
I20260926 20:49:09.216220 5480 ts_tablet_manager.cc:1397] T e4e9a3cec7d84d8896a97b1908d12240 P 93b6a47508be47669c5f292bb70a5ea8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 20:49:09.218004 5480 raft_consensus.cc:348] T e4e9a3cec7d84d8896a97b1908d12240 P 93b6a47508be47669c5f292bb70a5ea8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "027d6b7f71504dd28306d2962d794785" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38499 } } peers { permanent_uuid: "93b6a47508be47669c5f292bb70a5ea8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35073 } } peers { permanent_uuid: "562ae2267a4c4f789c46f00bfe367e4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36307 } }
I20260926 20:49:09.218156 5480 raft_consensus.cc:374] T e4e9a3cec7d84d8896a97b1908d12240 P 93b6a47508be47669c5f292bb70a5ea8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:49:09.218199 5480 raft_consensus.cc:729] T e4e9a3cec7d84d8896a97b1908d12240 P 93b6a47508be47669c5f292bb70a5ea8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93b6a47508be47669c5f292bb70a5ea8, State: Initialized, Role: FOLLOWER
I20260926 20:49:09.218116 5481 raft_consensus.cc:348] T e4e9a3cec7d84d8896a97b1908d12240 P 027d6b7f71504dd28306d2962d794785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "027d6b7f71504dd28306d2962d794785" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38499 } } peers { permanent_uuid: "93b6a47508be47669c5f292bb70a5ea8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35073 } } peers { permanent_uuid: "562ae2267a4c4f789c46f00bfe367e4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36307 } }
I20260926 20:49:09.218262 5481 raft_consensus.cc:374] T e4e9a3cec7d84d8896a97b1908d12240 P 027d6b7f71504dd28306d2962d794785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:49:09.218307 5481 raft_consensus.cc:729] T e4e9a3cec7d84d8896a97b1908d12240 P 027d6b7f71504dd28306d2962d794785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 027d6b7f71504dd28306d2962d794785, State: Initialized, Role: FOLLOWER
I20260926 20:49:09.218330 5480 consensus_queue.cc:260] T e4e9a3cec7d84d8896a97b1908d12240 P 93b6a47508be47669c5f292bb70a5ea8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "027d6b7f71504dd28306d2962d794785" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38499 } } peers { permanent_uuid: "93b6a47508be47669c5f292bb70a5ea8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35073 } } peers { permanent_uuid: "562ae2267a4c4f789c46f00bfe367e4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36307 } }
I20260926 20:49:09.218434 5481 consensus_queue.cc:260] T e4e9a3cec7d84d8896a97b1908d12240 P 027d6b7f71504dd28306d2962d794785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "027d6b7f71504dd28306d2962d794785" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38499 } } peers { permanent_uuid: "93b6a47508be47669c5f292bb70a5ea8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35073 } } peers { permanent_uuid: "562ae2267a4c4f789c46f00bfe367e4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36307 } }
I20260926 20:49:09.218588 5480 ts_tablet_manager.cc:1428] T e4e9a3cec7d84d8896a97b1908d12240 P 93b6a47508be47669c5f292bb70a5ea8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 20:49:09.218641 5335 heartbeater.cc:503] Master 127.0.98.62:45291 was elected leader, sending a full tablet report...
I20260926 20:49:09.218690 5481 ts_tablet_manager.cc:1428] T e4e9a3cec7d84d8896a97b1908d12240 P 027d6b7f71504dd28306d2962d794785: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:49:09.218844 5204 heartbeater.cc:503] Master 127.0.98.62:45291 was elected leader, sending a full tablet report...
I20260926 20:49:09.219126 5486 tablet_bootstrap.cc:492] T e4e9a3cec7d84d8896a97b1908d12240 P 562ae2267a4c4f789c46f00bfe367e4a: Bootstrap starting.
I20260926 20:49:09.219839 5486 tablet_bootstrap.cc:654] T e4e9a3cec7d84d8896a97b1908d12240 P 562ae2267a4c4f789c46f00bfe367e4a: Neither blocks nor log segments found. Creating new log.
I20260926 20:49:09.220188 5486 log.cc:824] T e4e9a3cec7d84d8896a97b1908d12240 P 562ae2267a4c4f789c46f00bfe367e4a: Log is configured to *not* fsync() on all Append() calls
I20260926 20:49:09.220862 5486 tablet_bootstrap.cc:492] T e4e9a3cec7d84d8896a97b1908d12240 P 562ae2267a4c4f789c46f00bfe367e4a: No bootstrap required, opened a new log
I20260926 20:49:09.220914 5486 ts_tablet_manager.cc:1397] T e4e9a3cec7d84d8896a97b1908d12240 P 562ae2267a4c4f789c46f00bfe367e4a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 20:49:09.222678 5486 raft_consensus.cc:348] T e4e9a3cec7d84d8896a97b1908d12240 P 562ae2267a4c4f789c46f00bfe367e4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "027d6b7f71504dd28306d2962d794785" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38499 } } peers { permanent_uuid: "93b6a47508be47669c5f292bb70a5ea8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35073 } } peers { permanent_uuid: "562ae2267a4c4f789c46f00bfe367e4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36307 } }
I20260926 20:49:09.222805 5486 raft_consensus.cc:374] T e4e9a3cec7d84d8896a97b1908d12240 P 562ae2267a4c4f789c46f00bfe367e4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:49:09.222826 5486 raft_consensus.cc:729] T e4e9a3cec7d84d8896a97b1908d12240 P 562ae2267a4c4f789c46f00bfe367e4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562ae2267a4c4f789c46f00bfe367e4a, State: Initialized, Role: FOLLOWER
I20260926 20:49:09.222957 5486 consensus_queue.cc:260] T e4e9a3cec7d84d8896a97b1908d12240 P 562ae2267a4c4f789c46f00bfe367e4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "027d6b7f71504dd28306d2962d794785" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38499 } } peers { permanent_uuid: "93b6a47508be47669c5f292bb70a5ea8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35073 } } peers { permanent_uuid: "562ae2267a4c4f789c46f00bfe367e4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36307 } }
I20260926 20:49:09.223218 5486 ts_tablet_manager.cc:1428] T e4e9a3cec7d84d8896a97b1908d12240 P 562ae2267a4c4f789c46f00bfe367e4a: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260926 20:49:09.223333 5466 heartbeater.cc:503] Master 127.0.98.62:45291 was elected leader, sending a full tablet report...
I20260926 20:49:09.267014 5485 raft_consensus.cc:482] T e4e9a3cec7d84d8896a97b1908d12240 P 027d6b7f71504dd28306d2962d794785 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:49:09.267136 5485 raft_consensus.cc:504] T e4e9a3cec7d84d8896a97b1908d12240 P 027d6b7f71504dd28306d2962d794785 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "027d6b7f71504dd28306d2962d794785" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38499 } } peers { permanent_uuid: "93b6a47508be47669c5f292bb70a5ea8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35073 } } peers { permanent_uuid: "562ae2267a4c4f789c46f00bfe367e4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36307 } }
I20260926 20:49:09.267551 5485 leader_election.cc:288] T e4e9a3cec7d84d8896a97b1908d12240 P 027d6b7f71504dd28306d2962d794785 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93b6a47508be47669c5f292bb70a5ea8 (127.0.98.2:35073), 562ae2267a4c4f789c46f00bfe367e4a (127.0.98.3:36307)
I20260926 20:49:09.268900 5206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:09.267764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35073 (local address 127.0.98.1:55405)
0926 20:49:09.267811 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:09.267824 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:09.267862 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:09.268027 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:09.268028 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:09.268031 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:09.268518 (+ 487us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:49:09.268519 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:09.268527 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:49:09.268541 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:09.268735 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:09.268735 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:09.268736 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 20:49:09.268805 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 20:49:09.268818 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260926 20:49:09.268934 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:09.267827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55405 (local address 127.0.98.2:35073)
0926 20:49:09.267867 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:09.267869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:09.267926 (+ 57us) server_negotiation.cc:407] Connection header received
0926 20:49:09.267953 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:09.267954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:09.267959 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:09.268006 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:09.268639 (+ 633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:09.268641 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:09.268641 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:49:09.268646 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:09.268665 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:49:09.268671 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:09.268732 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:09.268807 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 20:49:09.268821 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 20:49:09.269032 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:09.267888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36307 (local address 127.0.98.1:51263)
0926 20:49:09.268059 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:09.268067 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:09.268116 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:09.268267 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:09.268268 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:09.268271 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:09.268690 (+ 419us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:49:09.268691 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:09.268698 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:49:09.268710 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:09.268887 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:09.268888 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:09.268888 (+ 0us) client_negotiation.cc:769] Sending connection context
0926 20:49:09.268937 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 20:49:09.268950 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":114,"threads_started":1}
I20260926 20:49:09.269032 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:09.267975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51263 (local address 127.0.98.3:36307)
0926 20:49:09.268021 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:09.268023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:09.268131 (+ 108us) server_negotiation.cc:407] Connection header received
0926 20:49:09.268163 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:09.268164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:09.268170 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:09.268247 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:09.268793 (+ 546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:09.268794 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:09.268795 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:49:09.268799 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:09.268818 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:49:09.268823 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:09.268877 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:09.268938 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 20:49:09.268951 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 20:49:09.269310 5289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e9a3cec7d84d8896a97b1908d12240" candidate_uuid: "027d6b7f71504dd28306d2962d794785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93b6a47508be47669c5f292bb70a5ea8" is_pre_election: true
I20260926 20:49:09.269357 5420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e9a3cec7d84d8896a97b1908d12240" candidate_uuid: "027d6b7f71504dd28306d2962d794785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562ae2267a4c4f789c46f00bfe367e4a" is_pre_election: true
I20260926 20:49:09.269472 5289 raft_consensus.cc:2454] T e4e9a3cec7d84d8896a97b1908d12240 P 93b6a47508be47669c5f292bb70a5ea8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 027d6b7f71504dd28306d2962d794785 in term 0.
I20260926 20:49:09.269502 5420 raft_consensus.cc:2454] T e4e9a3cec7d84d8896a97b1908d12240 P 562ae2267a4c4f789c46f00bfe367e4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 027d6b7f71504dd28306d2962d794785 in term 0.
I20260926 20:49:09.269691 5092 leader_election.cc:302] T e4e9a3cec7d84d8896a97b1908d12240 P 027d6b7f71504dd28306d2962d794785 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 027d6b7f71504dd28306d2962d794785, 93b6a47508be47669c5f292bb70a5ea8; no voters:
I20260926 20:49:09.269845 5485 raft_consensus.cc:2781] T e4e9a3cec7d84d8896a97b1908d12240 P 027d6b7f71504dd28306d2962d794785 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:49:09.269920 5485 raft_consensus.cc:482] T e4e9a3cec7d84d8896a97b1908d12240 P 027d6b7f71504dd28306d2962d794785 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:49:09.269960 5485 raft_consensus.cc:3037] T e4e9a3cec7d84d8896a97b1908d12240 P 027d6b7f71504dd28306d2962d794785 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:49:09.270670 5485 raft_consensus.cc:504] T e4e9a3cec7d84d8896a97b1908d12240 P 027d6b7f71504dd28306d2962d794785 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "027d6b7f71504dd28306d2962d794785" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38499 } } peers { permanent_uuid: "93b6a47508be47669c5f292bb70a5ea8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35073 } } peers { permanent_uuid: "562ae2267a4c4f789c46f00bfe367e4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36307 } }
I20260926 20:49:09.270777 5485 leader_election.cc:288] T e4e9a3cec7d84d8896a97b1908d12240 P 027d6b7f71504dd28306d2962d794785 [CANDIDATE]: Term 1 election: Requested vote from peers 93b6a47508be47669c5f292bb70a5ea8 (127.0.98.2:35073), 562ae2267a4c4f789c46f00bfe367e4a (127.0.98.3:36307)
I20260926 20:49:09.270963 5289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e9a3cec7d84d8896a97b1908d12240" candidate_uuid: "027d6b7f71504dd28306d2962d794785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93b6a47508be47669c5f292bb70a5ea8"
I20260926 20:49:09.270982 5420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e9a3cec7d84d8896a97b1908d12240" candidate_uuid: "027d6b7f71504dd28306d2962d794785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562ae2267a4c4f789c46f00bfe367e4a"
I20260926 20:49:09.271051 5420 raft_consensus.cc:3037] T e4e9a3cec7d84d8896a97b1908d12240 P 562ae2267a4c4f789c46f00bfe367e4a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:49:09.271070 5289 raft_consensus.cc:3037] T e4e9a3cec7d84d8896a97b1908d12240 P 93b6a47508be47669c5f292bb70a5ea8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:49:09.271735 5420 raft_consensus.cc:2454] T e4e9a3cec7d84d8896a97b1908d12240 P 562ae2267a4c4f789c46f00bfe367e4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 027d6b7f71504dd28306d2962d794785 in term 1.
I20260926 20:49:09.271890 5289 raft_consensus.cc:2454] T e4e9a3cec7d84d8896a97b1908d12240 P 93b6a47508be47669c5f292bb70a5ea8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 027d6b7f71504dd28306d2962d794785 in term 1.
I20260926 20:49:09.271930 5092 leader_election.cc:302] T e4e9a3cec7d84d8896a97b1908d12240 P 027d6b7f71504dd28306d2962d794785 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 027d6b7f71504dd28306d2962d794785, 562ae2267a4c4f789c46f00bfe367e4a; no voters:
I20260926 20:49:09.272032 5485 raft_consensus.cc:2781] T e4e9a3cec7d84d8896a97b1908d12240 P 027d6b7f71504dd28306d2962d794785 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:49:09.272181 5485 raft_consensus.cc:686] T e4e9a3cec7d84d8896a97b1908d12240 P 027d6b7f71504dd28306d2962d794785 [term 1 LEADER]: Becoming Leader. State: Replica: 027d6b7f71504dd28306d2962d794785, State: Running, Role: LEADER
I20260926 20:49:09.272289 5485 consensus_queue.cc:237] T e4e9a3cec7d84d8896a97b1908d12240 P 027d6b7f71504dd28306d2962d794785 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "027d6b7f71504dd28306d2962d794785" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38499 } } peers { permanent_uuid: "93b6a47508be47669c5f292bb70a5ea8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35073 } } peers { permanent_uuid: "562ae2267a4c4f789c46f00bfe367e4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36307 } }
I20260926 20:49:09.273113 5015 catalog_manager.cc:5495] T e4e9a3cec7d84d8896a97b1908d12240 P 027d6b7f71504dd28306d2962d794785 reported cstate change: term changed from 0 to 1, leader changed from <none> to 027d6b7f71504dd28306d2962d794785 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "027d6b7f71504dd28306d2962d794785" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "027d6b7f71504dd28306d2962d794785" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93b6a47508be47669c5f292bb70a5ea8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562ae2267a4c4f789c46f00bfe367e4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36307 } health_report { overall_health: UNKNOWN } } }
I20260926 20:49:09.284488 5475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:09.283373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50512 (local address 127.0.98.1:38499)
0926 20:49:09.283438 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:09.283440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:09.283448 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:49:09.283530 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:09.283531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:09.283544 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:09.283594 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:09.284262 (+ 668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:09.284263 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:09.284264 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:49:09.284268 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:09.284283 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:49:09.284287 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:09.284340 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:09.284399 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 20:49:09.284412 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 20:49:09.286465 5420 raft_consensus.cc:1261] T e4e9a3cec7d84d8896a97b1908d12240 P 562ae2267a4c4f789c46f00bfe367e4a [term 1 FOLLOWER]: Refusing update from remote peer 027d6b7f71504dd28306d2962d794785: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:49:09.286464 5289 raft_consensus.cc:1261] T e4e9a3cec7d84d8896a97b1908d12240 P 93b6a47508be47669c5f292bb70a5ea8 [term 1 FOLLOWER]: Refusing update from remote peer 027d6b7f71504dd28306d2962d794785: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:49:09.286800 5490 consensus_queue.cc:1035] T e4e9a3cec7d84d8896a97b1908d12240 P 027d6b7f71504dd28306d2962d794785 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562ae2267a4c4f789c46f00bfe367e4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:49:09.286945 5485 consensus_queue.cc:1035] T e4e9a3cec7d84d8896a97b1908d12240 P 027d6b7f71504dd28306d2962d794785 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93b6a47508be47669c5f292bb70a5ea8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:49:09.289192 5493 mvcc.cc:205] Tried to move back new op lower bound from 7333706749074296832 to 7333706749019889664. Current Snapshot: MvccSnapshot[applied={T|T < 7333706749074296832}]
I20260926 20:49:09.289318 5494 mvcc.cc:205] Tried to move back new op lower bound from 7333706749074296832 to 7333706749019889664. Current Snapshot: MvccSnapshot[applied={T|T < 7333706749074296832}]
I20260926 20:49:09.289327 5495 mvcc.cc:205] Tried to move back new op lower bound from 7333706749074296832 to 7333706749019889664. Current Snapshot: MvccSnapshot[applied={T|T < 7333706749074296832}]
I20260926 20:49:09.291312 5015 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41230:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:49:09.291416 5015 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41230:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:49:09.292284 5015 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c64f1abc95b4ca5b18db6d789716b79: Sending DeleteTablet for 3 replicas of tablet e4e9a3cec7d84d8896a97b1908d12240
I20260926 20:49:09.292593 5138 tablet_service.cc:1514] Processing DeleteTablet for tablet e4e9a3cec7d84d8896a97b1908d12240 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:49:09 UTC) from {username='slave'} at 127.0.0.1:48708
I20260926 20:49:09.292649 5400 tablet_service.cc:1514] Processing DeleteTablet for tablet e4e9a3cec7d84d8896a97b1908d12240 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:49:09 UTC) from {username='slave'} at 127.0.0.1:38402
I20260926 20:49:09.292655 5269 tablet_service.cc:1514] Processing DeleteTablet for tablet e4e9a3cec7d84d8896a97b1908d12240 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:49:09 UTC) from {username='slave'} at 127.0.0.1:51666
I20260926 20:49:09.292878 5504 tablet_replica.cc:330] T e4e9a3cec7d84d8896a97b1908d12240 P 027d6b7f71504dd28306d2962d794785: stopping tablet replica
I20260926 20:49:09.292893 5505 tablet_replica.cc:330] T e4e9a3cec7d84d8896a97b1908d12240 P 93b6a47508be47669c5f292bb70a5ea8: stopping tablet replica
I20260926 20:49:09.292958 5503 tablet_replica.cc:330] T e4e9a3cec7d84d8896a97b1908d12240 P 562ae2267a4c4f789c46f00bfe367e4a: stopping tablet replica
I20260926 20:49:09.292982 5505 raft_consensus.cc:2229] T e4e9a3cec7d84d8896a97b1908d12240 P 93b6a47508be47669c5f292bb70a5ea8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:49:09.293011 5504 raft_consensus.cc:2229] T e4e9a3cec7d84d8896a97b1908d12240 P 027d6b7f71504dd28306d2962d794785 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:49:09.293056 5503 raft_consensus.cc:2229] T e4e9a3cec7d84d8896a97b1908d12240 P 562ae2267a4c4f789c46f00bfe367e4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:49:09.293107 5505 raft_consensus.cc:2258] T e4e9a3cec7d84d8896a97b1908d12240 P 93b6a47508be47669c5f292bb70a5ea8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:49:09.293120 5504 raft_consensus.cc:2258] T e4e9a3cec7d84d8896a97b1908d12240 P 027d6b7f71504dd28306d2962d794785 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:49:09.293112 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 5076
I20260926 20:49:09.293164 5503 raft_consensus.cc:2258] T e4e9a3cec7d84d8896a97b1908d12240 P 562ae2267a4c4f789c46f00bfe367e4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:49:09.294149 5505 ts_tablet_manager.cc:1905] T e4e9a3cec7d84d8896a97b1908d12240 P 93b6a47508be47669c5f292bb70a5ea8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:49:09.295655 5505 ts_tablet_manager.cc:1918] T e4e9a3cec7d84d8896a97b1908d12240 P 93b6a47508be47669c5f292bb70a5ea8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:49:09.295842 5505 log.cc:1196] T e4e9a3cec7d84d8896a97b1908d12240 P 93b6a47508be47669c5f292bb70a5ea8: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/ts-1/wal/wals/e4e9a3cec7d84d8896a97b1908d12240
I20260926 20:49:09.296181 5505 ts_tablet_manager.cc:1939] T e4e9a3cec7d84d8896a97b1908d12240 P 93b6a47508be47669c5f292bb70a5ea8: Deleting consensus metadata
I20260926 20:49:09.296388 5503 ts_tablet_manager.cc:1905] T e4e9a3cec7d84d8896a97b1908d12240 P 562ae2267a4c4f789c46f00bfe367e4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:49:09.296587 5001 catalog_manager.cc:4841] TS 93b6a47508be47669c5f292bb70a5ea8 (127.0.98.2:35073): tablet e4e9a3cec7d84d8896a97b1908d12240 (table test-table [id=1e3d38f0da844de6a7758dbdd8d15102]) successfully deleted
I20260926 20:49:09.298236 5503 ts_tablet_manager.cc:1918] T e4e9a3cec7d84d8896a97b1908d12240 P 562ae2267a4c4f789c46f00bfe367e4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:49:09.298310 5503 log.cc:1196] T e4e9a3cec7d84d8896a97b1908d12240 P 562ae2267a4c4f789c46f00bfe367e4a: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455725523895-392-0/minicluster-data/ts-2/wal/wals/e4e9a3cec7d84d8896a97b1908d12240
I20260926 20:49:09.298557 5503 ts_tablet_manager.cc:1939] T e4e9a3cec7d84d8896a97b1908d12240 P 562ae2267a4c4f789c46f00bfe367e4a: Deleting consensus metadata
I20260926 20:49:09.299185 5001 catalog_manager.cc:4841] TS 562ae2267a4c4f789c46f00bfe367e4a (127.0.98.3:36307): tablet e4e9a3cec7d84d8896a97b1908d12240 (table test-table [id=1e3d38f0da844de6a7758dbdd8d15102]) successfully deleted
W20260926 20:49:09.300108 5001 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38499 (error 108)
I20260926 20:49:09.300360 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 5207
I20260926 20:49:09.301088 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:09.300502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38499 (local address 127.0.0.1:50528)
0926 20:49:09.301025 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:09.301061 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495}
W20260926 20:49:09.301205 5001 catalog_manager.cc:4584] TS 027d6b7f71504dd28306d2962d794785 (127.0.98.1:38499): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e4e9a3cec7d84d8896a97b1908d12240: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38499: connect: Connection refused (error 111)
I20260926 20:49:09.305485 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 5338
I20260926 20:49:09.310081 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 4985
2026-09-26T20:49:09Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (646 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 20:49:09.329586 392 mini_oidc.cc:152] Starting JWKS server
I20260926 20:49:09.330575 392 webserver.cc:460] Webserver started at https://127.0.0.1:34951/ using document root <none> and password file <none>
I20260926 20:49:09.330648 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 20:49:09.330789 392 webserver.cc:460] Webserver started at http://127.0.0.1:40861/ using document root <none> and password file <none>
2026-09-26T20:49:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:49:09Z Disabled control of system clock
I20260926 20:49:09.338059 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34141
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34141
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34951/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455725523895-392-0/testchainca.cert with env {}
W20260926 20:49:09.418237 5512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:09.418452 5512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:09.418476 5512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:09.419714 5512 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:49:09.419821 5512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:09.419880 5512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:09.419899 5512 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:49:09.419924 5512 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 20:49:09.419955 5512 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 20:49:09.419975 5512 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34951/jwks.json
I20260926 20:49:09.421520 5512 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455725523895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34141
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34951/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455725523895-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:09.421797 5512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:09.422032 5512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:09.426766 5520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:09.426793 5517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:09.426774 5518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:09.426983 5512 server_base.cc:1034] running on GCE node
I20260926 20:49:09.427313 5512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:09.427610 5512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:09.428769 5512 hybrid_clock.cc:648] HybridClock initialized: now 1790455749428766 us; error 41 us; skew 500 ppm
I20260926 20:49:09.429934 5512 webserver.cc:460] Webserver started at http://127.0.98.62:34199/ using document root <none> and password file <none>
I20260926 20:49:09.430163 5512 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:09.430249 5512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:09.430395 5512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:49:09.431406 5512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455725523895-392-0/minicluster-data/master-0/data/instance:
uuid: "f4c8026fd8da4376952aecb55819da7c"
format_stamp: "Formatted at 2026-09-26 20:49:09 on dist-test-slave-hwlj"
server_key: "11cc37d235fe99c0176c84a291f9d91c"
server_key_iv: "5e5bfa12983068df8ade1f43fc220e0e"
server_key_version: "encryptionkey@0"
I20260926 20:49:09.431851 5512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455725523895-392-0/minicluster-data/master-0/wal/instance:
uuid: "f4c8026fd8da4376952aecb55819da7c"
format_stamp: "Formatted at 2026-09-26 20:49:09 on dist-test-slave-hwlj"
server_key: "11cc37d235fe99c0176c84a291f9d91c"
server_key_iv: "5e5bfa12983068df8ade1f43fc220e0e"
server_key_version: "encryptionkey@0"
I20260926 20:49:09.433089 5512 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:49:09.433802 5526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:09.433976 5512 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 20:49:09.434072 5512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "f4c8026fd8da4376952aecb55819da7c"
format_stamp: "Formatted at 2026-09-26 20:49:09 on dist-test-slave-hwlj"
server_key: "11cc37d235fe99c0176c84a291f9d91c"
server_key_iv: "5e5bfa12983068df8ade1f43fc220e0e"
server_key_version: "encryptionkey@0"
I20260926 20:49:09.434159 5512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:09.445547 5512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:09.445859 5512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:09.446017 5512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:09.449771 5512 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34141
I20260926 20:49:09.449813 5578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34141 every 8 connection(s)
I20260926 20:49:09.450151 5512 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:49:09.451143 5579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:49:09.452047 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 5512
I20260926 20:49:09.452158 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455725523895-392-0/minicluster-data/master-0/wal/instance
I20260926 20:49:09.452370 392 external_mini_cluster.cc:1428] Setting key 3be61df81fd4b3ea3d46ae88bbd3f336
I20260926 20:49:09.454222 5579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4c8026fd8da4376952aecb55819da7c: Bootstrap starting.
I20260926 20:49:09.454835 5579 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4c8026fd8da4376952aecb55819da7c: Neither blocks nor log segments found. Creating new log.
I20260926 20:49:09.455138 5579 log.cc:824] T 00000000000000000000000000000000 P f4c8026fd8da4376952aecb55819da7c: Log is configured to *not* fsync() on all Append() calls
I20260926 20:49:09.455893 5579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4c8026fd8da4376952aecb55819da7c: No bootstrap required, opened a new log
I20260926 20:49:09.457607 5582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:09.452847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57330 (local address 127.0.98.62:34141)
0926 20:49:09.453035 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:09.453043 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:09.453067 (+ 24us) server_negotiation.cc:407] Connection header received
0926 20:49:09.453449 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:09.453456 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:09.453513 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:09.453642 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:09.453841 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:09.454730 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:09.455816 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:09.456282 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:09.456370 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:09.456994 (+ 624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:09.456998 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:09.456998 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:49:09.457154 (+ 156us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:09.457183 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:49:09.457193 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:09.457241 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:09.457332 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 20:49:09.457389 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
I20260926 20:49:09.458185 5579 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4c8026fd8da4376952aecb55819da7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c8026fd8da4376952aecb55819da7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34141 } }
I20260926 20:49:09.458382 5579 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4c8026fd8da4376952aecb55819da7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:49:09.458429 5579 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4c8026fd8da4376952aecb55819da7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4c8026fd8da4376952aecb55819da7c, State: Initialized, Role: FOLLOWER
I20260926 20:49:09.458551 5579 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4c8026fd8da4376952aecb55819da7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c8026fd8da4376952aecb55819da7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34141 } }
I20260926 20:49:09.458599 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:49:09.458637 5579 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4c8026fd8da4376952aecb55819da7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:49:09.458683 5579 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4c8026fd8da4376952aecb55819da7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:49:09.458736 5579 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4c8026fd8da4376952aecb55819da7c [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:49:09.459372 5579 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4c8026fd8da4376952aecb55819da7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c8026fd8da4376952aecb55819da7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34141 } }
I20260926 20:49:09.459476 5579 leader_election.cc:302] T 00000000000000000000000000000000 P f4c8026fd8da4376952aecb55819da7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4c8026fd8da4376952aecb55819da7c; no voters:
I20260926 20:49:09.459620 5579 leader_election.cc:288] T 00000000000000000000000000000000 P f4c8026fd8da4376952aecb55819da7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:49:09.459692 5584 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4c8026fd8da4376952aecb55819da7c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:49:09.459848 5584 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4c8026fd8da4376952aecb55819da7c [term 1 LEADER]: Becoming Leader. State: Replica: f4c8026fd8da4376952aecb55819da7c, State: Running, Role: LEADER
I20260926 20:49:09.459897 5579 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4c8026fd8da4376952aecb55819da7c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:49:09.459939 5584 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4c8026fd8da4376952aecb55819da7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c8026fd8da4376952aecb55819da7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34141 } }
I20260926 20:49:09.460242 5585 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4c8026fd8da4376952aecb55819da7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4c8026fd8da4376952aecb55819da7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c8026fd8da4376952aecb55819da7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34141 } } }
I20260926 20:49:09.460253 5586 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4c8026fd8da4376952aecb55819da7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4c8026fd8da4376952aecb55819da7c. Latest consensus state: current_term: 1 leader_uuid: "f4c8026fd8da4376952aecb55819da7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c8026fd8da4376952aecb55819da7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34141 } } }
I20260926 20:49:09.460341 5585 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4c8026fd8da4376952aecb55819da7c [sys.catalog]: This master's current role is: LEADER
I20260926 20:49:09.460345 5586 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4c8026fd8da4376952aecb55819da7c [sys.catalog]: This master's current role is: LEADER
W20260926 20:49:09.461624 5600 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f4c8026fd8da4376952aecb55819da7c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:49:09.461666 5600 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:49:09.461725 5591 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:49:09.461855 5591 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:49:09.463178 5591 catalog_manager.cc:1347] Generated new cluster ID: 24333ee957254cc48aad773a68cfe1fe
I20260926 20:49:09.463219 5591 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:49:09.467070 5591 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:49:09.467559 5591 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:49:09.472812 5591 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4c8026fd8da4376952aecb55819da7c: Generated new TSK 0
I20260926 20:49:09.472965 5591 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 20:49:09.502322 5609 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 20:49:09.502480 5609 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 20:49:09.502671 5582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:09.479008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57332 (local address 127.0.98.62:34141)
0926 20:49:09.479067 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:09.479069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:09.479250 (+ 181us) server_negotiation.cc:407] Connection header received
0926 20:49:09.479340 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:09.479341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:09.479350 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:09.479408 (+ 58us) server_negotiation.cc:226] Negotiated authn=JWT
0926 20:49:09.479567 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:09.480114 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:09.481279 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:09.481462 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:09.481511 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:09.481727 (+ 216us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 20:49:09.502296 (+ 20569us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34951/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 20:49:09.502598 (+ 302us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57332: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34951/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":36}
W20260926 20:49:09.502733 5582 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57332: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34951/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 20:49:09.502930 5608 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:09.478919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34141 (local address 127.0.0.1:57332)
0926 20:49:09.479164 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:09.479175 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:09.479254 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:09.479434 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:09.479435 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:09.479437 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 20:49:09.479502 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:09.479505 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:09.480225 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:09.480226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:09.481197 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:09.481200 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:09.481530 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:09.481534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:09.481600 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:09.481623 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 20:49:09.502460 (+ 20837us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34951/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 20:49:09.502589 (+ 129us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:34141: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34951/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":106,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:34141: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34951/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 20:49:09.503417 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 5512
2026-09-26T20:49:09Z chronyd exiting
I20260926 20:49:09.538865 392 test_util.cc:175] -----------------------------------------------
I20260926 20:49:09.538954 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455725523895-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (210 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 20:49:09.539621 392 mini_oidc.cc:152] Starting JWKS server
I20260926 20:49:09.540328 392 webserver.cc:460] Webserver started at https://127.0.0.1:46137/ using document root <none> and password file <none>
I20260926 20:49:09.540365 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 20:49:09.540504 392 webserver.cc:460] Webserver started at http://127.0.0.1:36759/ using document root <none> and password file <none>
2026-09-26T20:49:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:49:09Z Disabled control of system clock
I20260926 20:49:09.547668 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40143
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40143
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46137/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455725523895-392-0/testchainca.cert with env {}
W20260926 20:49:09.626471 5615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:09.626677 5615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:09.626699 5615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:09.627832 5615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:49:09.627875 5615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:09.627888 5615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:09.627900 5615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:49:09.627912 5615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 20:49:09.627924 5615 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 20:49:09.627936 5615 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46137/jwks.json
I20260926 20:49:09.629384 5615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455725523895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40143
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46137/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455725523895-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:09.629571 5615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:09.629727 5615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:09.634745 5620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:09.634804 5621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:09.634895 5615 server_base.cc:1034] running on GCE node
W20260926 20:49:09.634745 5623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:09.635260 5615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:09.635512 5615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:09.636662 5615 hybrid_clock.cc:648] HybridClock initialized: now 1790455749636632 us; error 43 us; skew 500 ppm
I20260926 20:49:09.637913 5615 webserver.cc:460] Webserver started at http://127.0.98.62:33157/ using document root <none> and password file <none>
I20260926 20:49:09.638171 5615 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:09.638242 5615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:09.638388 5615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:49:09.639420 5615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455725523895-392-0/minicluster-data/master-0/data/instance:
uuid: "cbc9db0c267743fb9019b66ae832b632"
format_stamp: "Formatted at 2026-09-26 20:49:09 on dist-test-slave-hwlj"
server_key: "32919229c6e8b16623235a9a3c61cf95"
server_key_iv: "6a519a63333ec3631df4b90a9edc901a"
server_key_version: "encryptionkey@0"
I20260926 20:49:09.639846 5615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455725523895-392-0/minicluster-data/master-0/wal/instance:
uuid: "cbc9db0c267743fb9019b66ae832b632"
format_stamp: "Formatted at 2026-09-26 20:49:09 on dist-test-slave-hwlj"
server_key: "32919229c6e8b16623235a9a3c61cf95"
server_key_iv: "6a519a63333ec3631df4b90a9edc901a"
server_key_version: "encryptionkey@0"
I20260926 20:49:09.641196 5615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:49:09.641968 5629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:09.642151 5615 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:49:09.642248 5615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "cbc9db0c267743fb9019b66ae832b632"
format_stamp: "Formatted at 2026-09-26 20:49:09 on dist-test-slave-hwlj"
server_key: "32919229c6e8b16623235a9a3c61cf95"
server_key_iv: "6a519a63333ec3631df4b90a9edc901a"
server_key_version: "encryptionkey@0"
I20260926 20:49:09.642320 5615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:09.654685 5615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:09.654965 5615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:09.655117 5615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:09.658867 5615 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40143
I20260926 20:49:09.658948 5681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40143 every 8 connection(s)
I20260926 20:49:09.659262 5615 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:49:09.660243 5682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:49:09.661605 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 5615
I20260926 20:49:09.661723 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455725523895-392-0/minicluster-data/master-0/wal/instance
I20260926 20:49:09.661904 392 external_mini_cluster.cc:1428] Setting key 18bbb803ecc29b4c090970b0164be5bf
I20260926 20:49:09.662919 5682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbc9db0c267743fb9019b66ae832b632: Bootstrap starting.
I20260926 20:49:09.663626 5682 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbc9db0c267743fb9019b66ae832b632: Neither blocks nor log segments found. Creating new log.
I20260926 20:49:09.663962 5682 log.cc:824] T 00000000000000000000000000000000 P cbc9db0c267743fb9019b66ae832b632: Log is configured to *not* fsync() on all Append() calls
I20260926 20:49:09.664752 5682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbc9db0c267743fb9019b66ae832b632: No bootstrap required, opened a new log
I20260926 20:49:09.666929 5682 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbc9db0c267743fb9019b66ae832b632 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbc9db0c267743fb9019b66ae832b632" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40143 } }
I20260926 20:49:09.667060 5682 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbc9db0c267743fb9019b66ae832b632 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:49:09.667081 5682 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbc9db0c267743fb9019b66ae832b632 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbc9db0c267743fb9019b66ae832b632, State: Initialized, Role: FOLLOWER
I20260926 20:49:09.667238 5682 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbc9db0c267743fb9019b66ae832b632 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbc9db0c267743fb9019b66ae832b632" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40143 } }
I20260926 20:49:09.667323 5682 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbc9db0c267743fb9019b66ae832b632 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:49:09.667374 5682 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbc9db0c267743fb9019b66ae832b632 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:49:09.667424 5682 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbc9db0c267743fb9019b66ae832b632 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:49:09.667943 5685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:09.662404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53998 (local address 127.0.98.62:40143)
0926 20:49:09.662610 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:09.662619 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:09.662656 (+ 37us) server_negotiation.cc:407] Connection header received
0926 20:49:09.663181 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:09.663192 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:09.663285 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:09.663445 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:09.663610 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:09.664942 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:09.665752 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:09.666387 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:09.666531 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:09.667229 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:09.667234 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:09.667234 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:49:09.667449 (+ 215us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:09.667492 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:49:09.667506 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:09.667561 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:09.667635 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 20:49:09.667713 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":77,"threads_started":1}
I20260926 20:49:09.668119 5682 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbc9db0c267743fb9019b66ae832b632 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbc9db0c267743fb9019b66ae832b632" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40143 } }
I20260926 20:49:09.668386 5682 leader_election.cc:302] T 00000000000000000000000000000000 P cbc9db0c267743fb9019b66ae832b632 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbc9db0c267743fb9019b66ae832b632; no voters:
I20260926 20:49:09.668536 5682 leader_election.cc:288] T 00000000000000000000000000000000 P cbc9db0c267743fb9019b66ae832b632 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:49:09.668581 5687 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbc9db0c267743fb9019b66ae832b632 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:49:09.668694 5687 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbc9db0c267743fb9019b66ae832b632 [term 1 LEADER]: Becoming Leader. State: Replica: cbc9db0c267743fb9019b66ae832b632, State: Running, Role: LEADER
I20260926 20:49:09.668789 5682 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbc9db0c267743fb9019b66ae832b632 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:49:09.668793 5687 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbc9db0c267743fb9019b66ae832b632 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbc9db0c267743fb9019b66ae832b632" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40143 } }
I20260926 20:49:09.668944 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:49:09.669181 5688 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbc9db0c267743fb9019b66ae832b632 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc9db0c267743fb9019b66ae832b632. Latest consensus state: current_term: 1 leader_uuid: "cbc9db0c267743fb9019b66ae832b632" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbc9db0c267743fb9019b66ae832b632" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40143 } } }
I20260926 20:49:09.669275 5688 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbc9db0c267743fb9019b66ae832b632 [sys.catalog]: This master's current role is: LEADER
I20260926 20:49:09.669442 5689 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbc9db0c267743fb9019b66ae832b632 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbc9db0c267743fb9019b66ae832b632" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbc9db0c267743fb9019b66ae832b632" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40143 } } }
I20260926 20:49:09.669516 5689 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbc9db0c267743fb9019b66ae832b632 [sys.catalog]: This master's current role is: LEADER
I20260926 20:49:09.669481 5693 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:49:09.670094 5693 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:49:09.671618 5693 catalog_manager.cc:1347] Generated new cluster ID: 3a3689096a804140b639cdd78b29e9b9
I20260926 20:49:09.671669 5693 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:49:09.684432 5693 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:49:09.684922 5693 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:49:09.688239 5693 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbc9db0c267743fb9019b66ae832b632: Generated new TSK 0
I20260926 20:49:09.688422 5693 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 20:49:09.711627 5712 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 20:49:09.711719 5712 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 20:49:09.712137 5685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:09.697703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54012 (local address 127.0.98.62:40143)
0926 20:49:09.697776 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:09.697778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:09.697884 (+ 106us) server_negotiation.cc:407] Connection header received
0926 20:49:09.697963 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:09.697964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:09.697977 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:09.698049 (+ 72us) server_negotiation.cc:226] Negotiated authn=JWT
0926 20:49:09.698222 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:09.698762 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:09.699593 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:09.699878 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:09.699947 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:09.700133 (+ 186us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 20:49:09.711892 (+ 11759us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46137/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 20:49:09.712060 (+ 168us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54012: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46137/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":47}
W20260926 20:49:09.712198 5685 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54012: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46137/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260926 20:49:09.712230 5711 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:09.697644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40143 (local address 127.0.0.1:54012)
0926 20:49:09.697802 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:09.697813 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:09.697896 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:09.698049 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:09.698050 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:09.698051 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0926 20:49:09.698120 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:09.698122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:09.698859 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:09.698861 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:09.699497 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:09.699499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:09.699959 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:09.699963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:09.700022 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:09.700027 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 20:49:09.712019 (+ 11992us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46137/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 20:49:09.712116 (+ 97us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:40143: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46137/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":92,"threads_started":1}
I20260926 20:49:09.712438 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 5615
2026-09-26T20:49:09Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (210 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 20:49:09.750025 392 mini_oidc.cc:152] Starting JWKS server
I20260926 20:49:09.750671 392 webserver.cc:460] Webserver started at https://127.0.0.1:44111/ using document root <none> and password file <none>
I20260926 20:49:09.750713 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 20:49:09.750864 392 webserver.cc:460] Webserver started at http://127.0.0.1:34191/ using document root <none> and password file <none>
2026-09-26T20:49:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:49:09Z Disabled control of system clock
I20260926 20:49:09.758203 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34215
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34215
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44111/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 20:49:09.837900 5718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:09.838151 5718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:09.838191 5718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:09.839416 5718 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:49:09.839499 5718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:09.839521 5718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:09.839542 5718 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:49:09.839574 5718 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 20:49:09.839597 5718 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 20:49:09.839617 5718 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44111/jwks.json
I20260926 20:49:09.841117 5718 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790455725523895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34215
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44111/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:09.841410 5718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:09.841712 5718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:09.846702 5723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:09.846690 5724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:09.846797 5726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:09.847070 5718 server_base.cc:1034] running on GCE node
I20260926 20:49:09.847379 5718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:09.847718 5718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:09.848927 5718 hybrid_clock.cc:648] HybridClock initialized: now 1790455749848900 us; error 40 us; skew 500 ppm
I20260926 20:49:09.850294 5718 webserver.cc:460] Webserver started at http://127.0.98.62:38757/ using document root <none> and password file <none>
I20260926 20:49:09.850574 5718 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:09.850620 5718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:09.850797 5718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:49:09.851989 5718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790455725523895-392-0/minicluster-data/master-0/data/instance:
uuid: "93c729f7d42e4a4c8ca18be996f3e2e2"
format_stamp: "Formatted at 2026-09-26 20:49:09 on dist-test-slave-hwlj"
server_key: "2dcd0d18ad66f09b953aeff0fede4404"
server_key_iv: "c8841db56d1f121d2651ab9ad0ae139d"
server_key_version: "encryptionkey@0"
I20260926 20:49:09.852360 5718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790455725523895-392-0/minicluster-data/master-0/wal/instance:
uuid: "93c729f7d42e4a4c8ca18be996f3e2e2"
format_stamp: "Formatted at 2026-09-26 20:49:09 on dist-test-slave-hwlj"
server_key: "2dcd0d18ad66f09b953aeff0fede4404"
server_key_iv: "c8841db56d1f121d2651ab9ad0ae139d"
server_key_version: "encryptionkey@0"
I20260926 20:49:09.853669 5718 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:49:09.854452 5732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:09.854650 5718 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:49:09.854730 5718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "93c729f7d42e4a4c8ca18be996f3e2e2"
format_stamp: "Formatted at 2026-09-26 20:49:09 on dist-test-slave-hwlj"
server_key: "2dcd0d18ad66f09b953aeff0fede4404"
server_key_iv: "c8841db56d1f121d2651ab9ad0ae139d"
server_key_version: "encryptionkey@0"
I20260926 20:49:09.854784 5718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:09.868657 5718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:09.868911 5718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:09.869009 5718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:09.872601 5718 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34215
I20260926 20:49:09.872669 5784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34215 every 8 connection(s)
I20260926 20:49:09.872992 5718 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:49:09.873661 5785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:49:09.876341 5785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93c729f7d42e4a4c8ca18be996f3e2e2: Bootstrap starting.
I20260926 20:49:09.876907 5785 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93c729f7d42e4a4c8ca18be996f3e2e2: Neither blocks nor log segments found. Creating new log.
I20260926 20:49:09.877202 5785 log.cc:824] T 00000000000000000000000000000000 P 93c729f7d42e4a4c8ca18be996f3e2e2: Log is configured to *not* fsync() on all Append() calls
I20260926 20:49:09.877859 5785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93c729f7d42e4a4c8ca18be996f3e2e2: No bootstrap required, opened a new log
I20260926 20:49:09.879424 5785 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93c729f7d42e4a4c8ca18be996f3e2e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93c729f7d42e4a4c8ca18be996f3e2e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34215 } }
I20260926 20:49:09.879585 5785 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93c729f7d42e4a4c8ca18be996f3e2e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:49:09.879630 5785 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93c729f7d42e4a4c8ca18be996f3e2e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93c729f7d42e4a4c8ca18be996f3e2e2, State: Initialized, Role: FOLLOWER
I20260926 20:49:09.879860 5785 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93c729f7d42e4a4c8ca18be996f3e2e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93c729f7d42e4a4c8ca18be996f3e2e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34215 } }
I20260926 20:49:09.879945 5785 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93c729f7d42e4a4c8ca18be996f3e2e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:49:09.879992 5785 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93c729f7d42e4a4c8ca18be996f3e2e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:49:09.880040 5785 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93c729f7d42e4a4c8ca18be996f3e2e2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:49:09.880564 5785 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93c729f7d42e4a4c8ca18be996f3e2e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93c729f7d42e4a4c8ca18be996f3e2e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34215 } }
I20260926 20:49:09.880681 5785 leader_election.cc:302] T 00000000000000000000000000000000 P 93c729f7d42e4a4c8ca18be996f3e2e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93c729f7d42e4a4c8ca18be996f3e2e2; no voters:
I20260926 20:49:09.880861 5785 leader_election.cc:288] T 00000000000000000000000000000000 P 93c729f7d42e4a4c8ca18be996f3e2e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:49:09.880919 5788 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93c729f7d42e4a4c8ca18be996f3e2e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:49:09.881219 5788 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93c729f7d42e4a4c8ca18be996f3e2e2 [term 1 LEADER]: Becoming Leader. State: Replica: 93c729f7d42e4a4c8ca18be996f3e2e2, State: Running, Role: LEADER
I20260926 20:49:09.881331 5785 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93c729f7d42e4a4c8ca18be996f3e2e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:49:09.881331 5788 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93c729f7d42e4a4c8ca18be996f3e2e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93c729f7d42e4a4c8ca18be996f3e2e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34215 } }
I20260926 20:49:09.881793 5789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93c729f7d42e4a4c8ca18be996f3e2e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93c729f7d42e4a4c8ca18be996f3e2e2. Latest consensus state: current_term: 1 leader_uuid: "93c729f7d42e4a4c8ca18be996f3e2e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93c729f7d42e4a4c8ca18be996f3e2e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34215 } } }
I20260926 20:49:09.881892 5789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93c729f7d42e4a4c8ca18be996f3e2e2 [sys.catalog]: This master's current role is: LEADER
I20260926 20:49:09.881927 5790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93c729f7d42e4a4c8ca18be996f3e2e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93c729f7d42e4a4c8ca18be996f3e2e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93c729f7d42e4a4c8ca18be996f3e2e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34215 } } }
I20260926 20:49:09.881994 5790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93c729f7d42e4a4c8ca18be996f3e2e2 [sys.catalog]: This master's current role is: LEADER
I20260926 20:49:09.882311 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 5718
I20260926 20:49:09.882434 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790455725523895-392-0/minicluster-data/master-0/wal/instance
I20260926 20:49:09.882648 392 external_mini_cluster.cc:1428] Setting key 07e72732874cdab1bf10c5dad4f46e2e
W20260926 20:49:09.883270 5804 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 93c729f7d42e4a4c8ca18be996f3e2e2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:49:09.883348 5804 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:49:09.883410 5797 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:49:09.883648 5797 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:49:09.885321 5797 catalog_manager.cc:1347] Generated new cluster ID: 06c3c2552d8c4bbbab034df4cc05c8d2
I20260926 20:49:09.885383 5797 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:49:09.892148 5807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:09.883283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51952 (local address 127.0.98.62:34215)
0926 20:49:09.887923 (+ 4640us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:09.887931 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:09.887957 (+ 26us) server_negotiation.cc:407] Connection header received
0926 20:49:09.888050 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:09.888059 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:09.888112 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:09.888275 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:09.888489 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:09.889405 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:09.890209 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:09.890628 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:09.890770 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:09.891486 (+ 716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:09.891490 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:09.891490 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:49:09.891669 (+ 179us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:09.891701 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:49:09.891712 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:09.891811 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:09.891883 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 20:49:09.891963 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4557,"thread_start_us":89,"threads_started":1}
I20260926 20:49:09.892988 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:49:09.900779 5797 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:49:09.901271 5797 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:49:09.908593 5797 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93c729f7d42e4a4c8ca18be996f3e2e2: Generated new TSK 0
I20260926 20:49:09.908780 5797 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:49:09.920374 5807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:09.916422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51962 (local address 127.0.98.62:34215)
0926 20:49:09.916492 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:09.916495 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:09.916585 (+ 90us) server_negotiation.cc:407] Connection header received
0926 20:49:09.916640 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:09.916641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:09.916673 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:09.916746 (+ 73us) server_negotiation.cc:226] Negotiated authn=JWT
0926 20:49:09.916927 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:09.917699 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:09.918578 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:09.918904 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:09.918975 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:09.919148 (+ 173us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 20:49:09.920166 (+ 1018us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44111/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 20:49:09.920302 (+ 136us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51962: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44111/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":44}
W20260926 20:49:09.920439 5807 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51962: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44111/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260926 20:49:09.922379 5814 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:09.916333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34215 (local address 127.0.0.1:51962)
0926 20:49:09.916503 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:09.916514 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:09.916576 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:09.916781 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:09.916783 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:09.916785 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 20:49:09.916841 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:09.916843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:09.917797 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:09.917798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:09.918485 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:09.918487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:09.918981 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:09.918988 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:09.919031 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:09.919036 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 20:49:09.920270 (+ 1234us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44111/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 20:49:09.922269 (+ 1999us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:34215: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44111/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":1791,"thread_start_us":102,"threads_started":1}
I20260926 20:49:09.922601 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 5718
E20260926 20:49:09.931792 5815 webserver.cc:533] Webserver: sslize failed:
E20260926 20:49:09.931869 5815 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-26T20:49:09Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (210 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T20:49:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:49:09Z Disabled control of system clock
I20260926 20:49:09.968361 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40673
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40673
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455725523895-392-0/testchainca.cert with env {}
W20260926 20:49:10.047364 5819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:10.047565 5819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:10.047587 5819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:10.048692 5819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:49:10.048731 5819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:10.048744 5819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:10.048758 5819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:49:10.048770 5819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 20:49:10.048782 5819 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 20:49:10.048794 5819 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 20:49:10.050185 5819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455725523895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40673
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455725523895-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:10.050377 5819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:10.050541 5819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:10.055400 5827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:10.055406 5825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:10.055418 5819 server_base.cc:1034] running on GCE node
W20260926 20:49:10.055395 5824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:10.056010 5819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:10.056324 5819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:10.057508 5819 hybrid_clock.cc:648] HybridClock initialized: now 1790455750057498 us; error 37 us; skew 500 ppm
I20260926 20:49:10.058746 5819 webserver.cc:460] Webserver started at http://127.0.98.62:46017/ using document root <none> and password file <none>
I20260926 20:49:10.059005 5819 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:10.059077 5819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:10.059229 5819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:49:10.060384 5819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455725523895-392-0/minicluster-data/master-0/data/instance:
uuid: "a89b419d827646848dc1abc288e3e07e"
format_stamp: "Formatted at 2026-09-26 20:49:10 on dist-test-slave-hwlj"
server_key: "a1b88cebcad512cfc96de9cf77141e2d"
server_key_iv: "a439439ea41253eb8fd8d07be8628ca2"
server_key_version: "encryptionkey@0"
I20260926 20:49:10.060781 5819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455725523895-392-0/minicluster-data/master-0/wal/instance:
uuid: "a89b419d827646848dc1abc288e3e07e"
format_stamp: "Formatted at 2026-09-26 20:49:10 on dist-test-slave-hwlj"
server_key: "a1b88cebcad512cfc96de9cf77141e2d"
server_key_iv: "a439439ea41253eb8fd8d07be8628ca2"
server_key_version: "encryptionkey@0"
I20260926 20:49:10.062052 5819 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:49:10.062750 5833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:10.062927 5819 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:49:10.063023 5819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "a89b419d827646848dc1abc288e3e07e"
format_stamp: "Formatted at 2026-09-26 20:49:10 on dist-test-slave-hwlj"
server_key: "a1b88cebcad512cfc96de9cf77141e2d"
server_key_iv: "a439439ea41253eb8fd8d07be8628ca2"
server_key_version: "encryptionkey@0"
I20260926 20:49:10.063107 5819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:10.077596 5819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:10.077903 5819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:10.078054 5819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:10.081810 5819 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40673
I20260926 20:49:10.081898 5885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40673 every 8 connection(s)
I20260926 20:49:10.082227 5819 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:49:10.082701 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 5819
I20260926 20:49:10.082819 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455725523895-392-0/minicluster-data/master-0/wal/instance
I20260926 20:49:10.083022 392 external_mini_cluster.cc:1428] Setting key 8b92a6c1e0ff38e5e347c3e55d3e3407
I20260926 20:49:10.083516 5886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:49:10.086875 5886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a89b419d827646848dc1abc288e3e07e: Bootstrap starting.
I20260926 20:49:10.087632 5886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a89b419d827646848dc1abc288e3e07e: Neither blocks nor log segments found. Creating new log.
I20260926 20:49:10.087980 5886 log.cc:824] T 00000000000000000000000000000000 P a89b419d827646848dc1abc288e3e07e: Log is configured to *not* fsync() on all Append() calls
I20260926 20:49:10.088842 5886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a89b419d827646848dc1abc288e3e07e: No bootstrap required, opened a new log
I20260926 20:49:10.089293 5889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:10.083569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34198 (local address 127.0.98.62:40673)
0926 20:49:10.083827 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:10.083837 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:10.083869 (+ 32us) server_negotiation.cc:407] Connection header received
0926 20:49:10.084400 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:10.084409 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:10.084496 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:10.084649 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:10.084855 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:10.086208 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:10.087035 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:10.087666 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:10.087821 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:10.088517 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:10.088521 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:10.088521 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:49:10.088727 (+ 206us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:10.088767 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:49:10.088783 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:10.088853 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:10.088982 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 20:49:10.089058 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":104,"threads_started":1}
I20260926 20:49:10.090335 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:49:10.091316 5886 raft_consensus.cc:348] T 00000000000000000000000000000000 P a89b419d827646848dc1abc288e3e07e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a89b419d827646848dc1abc288e3e07e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40673 } }
I20260926 20:49:10.091480 5886 raft_consensus.cc:374] T 00000000000000000000000000000000 P a89b419d827646848dc1abc288e3e07e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:49:10.091516 5886 raft_consensus.cc:729] T 00000000000000000000000000000000 P a89b419d827646848dc1abc288e3e07e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a89b419d827646848dc1abc288e3e07e, State: Initialized, Role: FOLLOWER
I20260926 20:49:10.091635 5886 consensus_queue.cc:260] T 00000000000000000000000000000000 P a89b419d827646848dc1abc288e3e07e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a89b419d827646848dc1abc288e3e07e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40673 } }
I20260926 20:49:10.091710 5886 raft_consensus.cc:388] T 00000000000000000000000000000000 P a89b419d827646848dc1abc288e3e07e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:49:10.091732 5886 raft_consensus.cc:482] T 00000000000000000000000000000000 P a89b419d827646848dc1abc288e3e07e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:49:10.091825 5886 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a89b419d827646848dc1abc288e3e07e [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:49:10.092324 5886 raft_consensus.cc:504] T 00000000000000000000000000000000 P a89b419d827646848dc1abc288e3e07e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a89b419d827646848dc1abc288e3e07e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40673 } }
I20260926 20:49:10.092401 5886 leader_election.cc:302] T 00000000000000000000000000000000 P a89b419d827646848dc1abc288e3e07e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a89b419d827646848dc1abc288e3e07e; no voters:
I20260926 20:49:10.092516 5886 leader_election.cc:288] T 00000000000000000000000000000000 P a89b419d827646848dc1abc288e3e07e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:49:10.092598 5891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a89b419d827646848dc1abc288e3e07e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:49:10.092737 5891 raft_consensus.cc:686] T 00000000000000000000000000000000 P a89b419d827646848dc1abc288e3e07e [term 1 LEADER]: Becoming Leader. State: Replica: a89b419d827646848dc1abc288e3e07e, State: Running, Role: LEADER
I20260926 20:49:10.092773 5886 sys_catalog.cc:564] T 00000000000000000000000000000000 P a89b419d827646848dc1abc288e3e07e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:49:10.092818 5891 consensus_queue.cc:237] T 00000000000000000000000000000000 P a89b419d827646848dc1abc288e3e07e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a89b419d827646848dc1abc288e3e07e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40673 } }
I20260926 20:49:10.093199 5892 sys_catalog.cc:455] T 00000000000000000000000000000000 P a89b419d827646848dc1abc288e3e07e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a89b419d827646848dc1abc288e3e07e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a89b419d827646848dc1abc288e3e07e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40673 } } }
I20260926 20:49:10.093204 5893 sys_catalog.cc:455] T 00000000000000000000000000000000 P a89b419d827646848dc1abc288e3e07e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a89b419d827646848dc1abc288e3e07e. Latest consensus state: current_term: 1 leader_uuid: "a89b419d827646848dc1abc288e3e07e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a89b419d827646848dc1abc288e3e07e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40673 } } }
I20260926 20:49:10.093320 5892 sys_catalog.cc:458] T 00000000000000000000000000000000 P a89b419d827646848dc1abc288e3e07e [sys.catalog]: This master's current role is: LEADER
I20260926 20:49:10.093327 5893 sys_catalog.cc:458] T 00000000000000000000000000000000 P a89b419d827646848dc1abc288e3e07e [sys.catalog]: This master's current role is: LEADER
I20260926 20:49:10.093544 5899 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:49:10.094048 5899 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:49:10.095503 5899 catalog_manager.cc:1347] Generated new cluster ID: 7b52e4aeb137486085d6f979afa4aaf9
I20260926 20:49:10.095559 5899 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:49:10.110709 5899 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:49:10.111403 5899 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:49:10.115371 5899 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a89b419d827646848dc1abc288e3e07e: Generated new TSK 0
I20260926 20:49:10.115586 5899 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:49:10.598827 5889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:10.128300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34204 (local address 127.0.98.62:40673)
0926 20:49:10.128362 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:10.128364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:10.128471 (+ 107us) server_negotiation.cc:407] Connection header received
0926 20:49:10.128538 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:10.128539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:10.128550 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:10.128609 (+ 59us) server_negotiation.cc:226] Negotiated authn=JWT
0926 20:49:10.128790 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:10.129313 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:10.130454 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:10.130652 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:10.130702 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:10.130953 (+ 251us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 20:49:10.598601 (+467648us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 20:49:10.598762 (+ 161us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34204: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":41}
W20260926 20:49:10.598922 5889 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34204: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 20:49:10.599071 5915 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:10.128200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40673 (local address 127.0.0.1:34204)
0926 20:49:10.128390 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:10.128402 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:10.128471 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:10.128630 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:10.128631 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:10.128632 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0926 20:49:10.128697 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:10.128700 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:10.129396 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:10.129397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:10.130366 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:10.130369 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:10.130736 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:10.130742 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:10.130804 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:10.130809 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 20:49:10.598797 (+467988us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 20:49:10.598923 (+ 126us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:40673: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":120,"threads_started":1}
I20260926 20:49:10.599646 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 5819
2026-09-26T20:49:10Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (657 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (85 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (86 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:49:10 dist-test-slave-hwlj krb5kdc[5927](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:49:10 dist-test-slave-hwlj krb5kdc[5927](info): set up 2 sockets
Sep 26 20:49:10 dist-test-slave-hwlj krb5kdc[5927](info): commencing operation
krb5kdc: starting...
W20260926 20:49:11.800009 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/krb5kdc/test-user.keytab.
Sep 26 20:49:11 dist-test-slave-hwlj krb5kdc[5927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:49:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:49:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 20:49:11.918905 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45621
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45621
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:49:11.999447 5940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:11.999712 5940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:11.999785 5940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:12.000988 5940 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:49:12.001056 5940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:12.001080 5940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:12.001101 5940 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:49:12.001132 5940 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:49:12.002701 5940 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:12.002981 5940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:12.003217 5940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:12.008088 5948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:12.008078 5946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:12.008093 5945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:12.008352 5940 server_base.cc:1034] running on GCE node
I20260926 20:49:12.008615 5940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:12.008880 5940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:12.010025 5940 hybrid_clock.cc:648] HybridClock initialized: now 1790455752010013 us; error 36 us; skew 500 ppm
Sep 26 20:49:12 dist-test-slave-hwlj krb5kdc[5927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:49:12.012435 5940 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:49:12.012898 5940 webserver.cc:460] Webserver started at http://127.0.98.62:32943/ using document root <none> and password file <none>
I20260926 20:49:12.013094 5940 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:12.013136 5940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:12.013262 5940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:49:12.014343 5940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/master-0/data/instance:
uuid: "611296bd402b4495a047a5a132b62331"
format_stamp: "Formatted at 2026-09-26 20:49:12 on dist-test-slave-hwlj"
server_key: "2aba2d4c6f277bb8fff6802d5c5ab872"
server_key_iv: "88e21bb030336a964e5bef8a031a7de7"
server_key_version: "encryptionkey@0"
I20260926 20:49:12.014664 5940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/master-0/wal/instance:
uuid: "611296bd402b4495a047a5a132b62331"
format_stamp: "Formatted at 2026-09-26 20:49:12 on dist-test-slave-hwlj"
server_key: "2aba2d4c6f277bb8fff6802d5c5ab872"
server_key_iv: "88e21bb030336a964e5bef8a031a7de7"
server_key_version: "encryptionkey@0"
I20260926 20:49:12.016013 5940 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:49:12.016759 5955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:12.016937 5940 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:49:12.017045 5940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "611296bd402b4495a047a5a132b62331"
format_stamp: "Formatted at 2026-09-26 20:49:12 on dist-test-slave-hwlj"
server_key: "2aba2d4c6f277bb8fff6802d5c5ab872"
server_key_iv: "88e21bb030336a964e5bef8a031a7de7"
server_key_version: "encryptionkey@0"
I20260926 20:49:12.017118 5940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:12.031424 5940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:12.033815 5940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:12.033979 5940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:12.038141 5940 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45621
I20260926 20:49:12.038183 6007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45621 every 8 connection(s)
I20260926 20:49:12.038726 5940 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:49:12.039408 6008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:49:12.041693 6008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 611296bd402b4495a047a5a132b62331: Bootstrap starting.
I20260926 20:49:12.042273 6008 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 611296bd402b4495a047a5a132b62331: Neither blocks nor log segments found. Creating new log.
I20260926 20:49:12.042560 6008 log.cc:824] T 00000000000000000000000000000000 P 611296bd402b4495a047a5a132b62331: Log is configured to *not* fsync() on all Append() calls
I20260926 20:49:12.043233 6008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 611296bd402b4495a047a5a132b62331: No bootstrap required, opened a new log
I20260926 20:49:12.043841 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 5940
I20260926 20:49:12.043955 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/master-0/wal/instance
I20260926 20:49:12.044138 392 external_mini_cluster.cc:1428] Setting key 00900766450d5192d5dcaa0776709258
I20260926 20:49:12.045248 6008 raft_consensus.cc:348] T 00000000000000000000000000000000 P 611296bd402b4495a047a5a132b62331 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "611296bd402b4495a047a5a132b62331" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45621 } }
I20260926 20:49:12.045413 6008 raft_consensus.cc:374] T 00000000000000000000000000000000 P 611296bd402b4495a047a5a132b62331 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:49:12.045450 6008 raft_consensus.cc:729] T 00000000000000000000000000000000 P 611296bd402b4495a047a5a132b62331 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 611296bd402b4495a047a5a132b62331, State: Initialized, Role: FOLLOWER
I20260926 20:49:12.045596 6008 consensus_queue.cc:260] T 00000000000000000000000000000000 P 611296bd402b4495a047a5a132b62331 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "611296bd402b4495a047a5a132b62331" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45621 } }
I20260926 20:49:12.045694 6008 raft_consensus.cc:388] T 00000000000000000000000000000000 P 611296bd402b4495a047a5a132b62331 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:49:12.045732 6008 raft_consensus.cc:482] T 00000000000000000000000000000000 P 611296bd402b4495a047a5a132b62331 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:49:12.045796 6008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 611296bd402b4495a047a5a132b62331 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:49:12.046439 6008 raft_consensus.cc:504] T 00000000000000000000000000000000 P 611296bd402b4495a047a5a132b62331 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "611296bd402b4495a047a5a132b62331" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45621 } }
I20260926 20:49:12.046521 6008 leader_election.cc:302] T 00000000000000000000000000000000 P 611296bd402b4495a047a5a132b62331 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 611296bd402b4495a047a5a132b62331; no voters:
I20260926 20:49:12.046715 6008 leader_election.cc:288] T 00000000000000000000000000000000 P 611296bd402b4495a047a5a132b62331 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:49:12.046766 6013 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 611296bd402b4495a047a5a132b62331 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:49:12.046895 6013 raft_consensus.cc:686] T 00000000000000000000000000000000 P 611296bd402b4495a047a5a132b62331 [term 1 LEADER]: Becoming Leader. State: Replica: 611296bd402b4495a047a5a132b62331, State: Running, Role: LEADER
I20260926 20:49:12.047000 6008 sys_catalog.cc:564] T 00000000000000000000000000000000 P 611296bd402b4495a047a5a132b62331 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:49:12.046973 6013 consensus_queue.cc:237] T 00000000000000000000000000000000 P 611296bd402b4495a047a5a132b62331 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "611296bd402b4495a047a5a132b62331" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45621 } }
I20260926 20:49:12.047328 6013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 611296bd402b4495a047a5a132b62331 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 611296bd402b4495a047a5a132b62331. Latest consensus state: current_term: 1 leader_uuid: "611296bd402b4495a047a5a132b62331" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "611296bd402b4495a047a5a132b62331" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45621 } } }
I20260926 20:49:12.047410 6013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 611296bd402b4495a047a5a132b62331 [sys.catalog]: This master's current role is: LEADER
I20260926 20:49:12.047340 6015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 611296bd402b4495a047a5a132b62331 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "611296bd402b4495a047a5a132b62331" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "611296bd402b4495a047a5a132b62331" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45621 } } }
I20260926 20:49:12.047574 6015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 611296bd402b4495a047a5a132b62331 [sys.catalog]: This master's current role is: LEADER
I20260926 20:49:12.047613 6019 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:49:12.048106 6019 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:49:12.049788 6019 catalog_manager.cc:1347] Generated new cluster ID: 4e1bbfa4fcf647bebe986b6c8ce1b6f2
I20260926 20:49:12.049841 6019 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 20:49:12 dist-test-slave-hwlj krb5kdc[5927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:49:12.062383 6019 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:49:12.063002 6019 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:49:12.065505 6012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:12.044678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42518 (local address 127.0.98.62:45621)
0926 20:49:12.044845 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:12.044853 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:12.044878 (+ 25us) server_negotiation.cc:407] Connection header received
0926 20:49:12.045243 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:12.045249 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:12.045293 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:12.045400 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:12.046210 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:12.047497 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:12.048563 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:12.049162 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:12.049595 (+ 433us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:12.061839 (+ 12244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:12.061844 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:12.061845 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:12.061857 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:12.063900 (+ 2043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:12.064283 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:12.064286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:12.064287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:12.064406 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:12.064605 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:12.064606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:12.064606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:12.064893 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:12.065038 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:12.065137 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 20:49:12.065284 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":84,"threads_started":1}
I20260926 20:49:12.068816 6019 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 611296bd402b4495a047a5a132b62331: Generated new TSK 0
I20260926 20:49:12.068979 6019 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 20:49:12.094715 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45621
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:49:12.174881 6034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:12.175084 6034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:12.175108 6034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:12.176276 6034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:12.176318 6034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:12.176376 6034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:49:12.177852 6034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:12.178107 6034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:12.178331 6034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:12.183370 6042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:12.183337 6039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:12.183362 6040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:12.183347 6034 server_base.cc:1034] running on GCE node
I20260926 20:49:12.183846 6034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:12.184100 6034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:12.185238 6034 hybrid_clock.cc:648] HybridClock initialized: now 1790455752185228 us; error 29 us; skew 500 ppm
Sep 26 20:49:12 dist-test-slave-hwlj krb5kdc[5927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:49:12.187727 6034 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 20:49:12.188225 6034 webserver.cc:460] Webserver started at http://127.0.98.1:43763/ using document root <none> and password file <none>
I20260926 20:49:12.188473 6034 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:12.188521 6034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:12.188684 6034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:49:12.189793 6034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/ts-0/data/instance:
uuid: "791c9cc9656b477d90e4537ab595af36"
format_stamp: "Formatted at 2026-09-26 20:49:12 on dist-test-slave-hwlj"
server_key: "81febaf1970bce48c2ad1c35f0ab897a"
server_key_iv: "287ea81805745a61808bcdcd6b402a2e"
server_key_version: "encryptionkey@0"
I20260926 20:49:12.190174 6034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/ts-0/wal/instance:
uuid: "791c9cc9656b477d90e4537ab595af36"
format_stamp: "Formatted at 2026-09-26 20:49:12 on dist-test-slave-hwlj"
server_key: "81febaf1970bce48c2ad1c35f0ab897a"
server_key_iv: "287ea81805745a61808bcdcd6b402a2e"
server_key_version: "encryptionkey@0"
I20260926 20:49:12.191543 6034 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 20:49:12.192433 6049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:12.192611 6034 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:49:12.192713 6034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/ts-0/wal
uuid: "791c9cc9656b477d90e4537ab595af36"
format_stamp: "Formatted at 2026-09-26 20:49:12 on dist-test-slave-hwlj"
server_key: "81febaf1970bce48c2ad1c35f0ab897a"
server_key_iv: "287ea81805745a61808bcdcd6b402a2e"
server_key_version: "encryptionkey@0"
I20260926 20:49:12.192792 6034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:12.205892 6034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:12.208349 6034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:12.208518 6034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:12.208796 6034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:49:12.209195 6034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:49:12.209260 6034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:12.209300 6034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:49:12.209329 6034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:12.215433 6034 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43253
I20260926 20:49:12.215517 6162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43253 every 8 connection(s)
I20260926 20:49:12.215878 6034 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:49:12.219435 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 6034
I20260926 20:49:12.219525 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/ts-0/wal/instance
I20260926 20:49:12.219915 392 external_mini_cluster.cc:1428] Setting key abd490dbbd21e462e887361fda81a350
Sep 26 20:49:12 dist-test-slave-hwlj krb5kdc[5927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:49:12.226923 6012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:12.216678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52237 (local address 127.0.98.62:45621)
0926 20:49:12.216764 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:12.216767 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:12.216969 (+ 202us) server_negotiation.cc:407] Connection header received
0926 20:49:12.217934 (+ 965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:12.217935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:12.217944 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:12.218002 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:12.218607 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:12.219097 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:12.220488 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:12.220730 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:12.220826 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:12.223339 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:12.223341 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:12.223341 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:12.223348 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:12.225244 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:12.225842 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:12.225843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:12.225844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:12.225913 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:12.226179 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:12.226181 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:12.226181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:12.226350 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:12.226425 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:12.226705 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 20:49:12.226817 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 20:49:12.227053 6165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:12.216648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45621 (local address 127.0.98.1:52237)
0926 20:49:12.216833 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:12.216876 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:12.217843 (+ 967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:12.218074 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:12.218077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:12.218342 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:12.218531 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:12.218534 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:12.219209 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:12.219209 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:12.220412 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:12.220414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:12.220846 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:12.220847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:12.221002 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:12.221600 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:12.221609 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:12.223236 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:12.225378 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:12.225385 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:12.225387 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:12.225752 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:12.225991 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:12.225992 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:12.225992 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:12.226109 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:12.226423 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:12.226427 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:12.226560 (+ 133us) client_negotiation.cc:769] Sending connection context
0926 20:49:12.226652 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 20:49:12.226819 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":104,"threads_started":1}
I20260926 20:49:12.227648 6163 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45621
I20260926 20:49:12.227810 6163 heartbeater.cc:464] Registering TS with master...
I20260926 20:49:12.228019 6163 heartbeater.cc:511] Master 127.0.98.62:45621 requested a full tablet report, sending...
I20260926 20:49:12.228682 5972 ts_manager.cc:194] Registered new tserver with Master: 791c9cc9656b477d90e4537ab595af36 (127.0.98.1:43253)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 20:49:12.229583 5972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52237
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 20:49:12.246459 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45621
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:49:12.327728 6168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:12.327982 6168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:12.328017 6168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:12.329140 6168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:12.329206 6168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:12.329250 6168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:49:12.330685 6168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:12.330942 6168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:12.331164 6168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:12.336040 6176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:12.336084 6174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:12.336036 6173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:12.336390 6168 server_base.cc:1034] running on GCE node
I20260926 20:49:12.336659 6168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:12.336905 6168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:12.338078 6168 hybrid_clock.cc:648] HybridClock initialized: now 1790455752338077 us; error 43 us; skew 500 ppm
Sep 26 20:49:12 dist-test-slave-hwlj krb5kdc[5927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:49:12.340591 6168 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 20:49:12.341034 6168 webserver.cc:460] Webserver started at http://127.0.98.2:33239/ using document root <none> and password file <none>
I20260926 20:49:12.341259 6168 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:12.341327 6168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:12.341475 6168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:49:12.342512 6168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/ts-1/data/instance:
uuid: "5d69aba77175443bbcf9cda9c8e6dd04"
format_stamp: "Formatted at 2026-09-26 20:49:12 on dist-test-slave-hwlj"
server_key: "7aa92d33380830326f2f51ce97323917"
server_key_iv: "ee7a1d0334841b0e25275251810c6a31"
server_key_version: "encryptionkey@0"
I20260926 20:49:12.342866 6168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5d69aba77175443bbcf9cda9c8e6dd04"
format_stamp: "Formatted at 2026-09-26 20:49:12 on dist-test-slave-hwlj"
server_key: "7aa92d33380830326f2f51ce97323917"
server_key_iv: "ee7a1d0334841b0e25275251810c6a31"
server_key_version: "encryptionkey@0"
I20260926 20:49:12.344250 6168 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:49:12.345096 6183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:12.345276 6168 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:49:12.345396 6168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/ts-1/wal
uuid: "5d69aba77175443bbcf9cda9c8e6dd04"
format_stamp: "Formatted at 2026-09-26 20:49:12 on dist-test-slave-hwlj"
server_key: "7aa92d33380830326f2f51ce97323917"
server_key_iv: "ee7a1d0334841b0e25275251810c6a31"
server_key_version: "encryptionkey@0"
I20260926 20:49:12.345482 6168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:12.367408 6168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:12.369294 6168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:12.369459 6168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:12.369709 6168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:49:12.370013 6168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:49:12.370074 6168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:12.370121 6168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:49:12.370146 6168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:12.376125 6168 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33109
I20260926 20:49:12.376230 6296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33109 every 8 connection(s)
I20260926 20:49:12.376546 6168 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
I20260926 20:49:12.381654 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 6168
I20260926 20:49:12.381764 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/ts-1/wal/instance
I20260926 20:49:12.381948 392 external_mini_cluster.cc:1428] Setting key 5083071912221a1845057be4bd18133d
Sep 26 20:49:12 dist-test-slave-hwlj krb5kdc[5927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:49:12.386523 6012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:12.377352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52107 (local address 127.0.98.62:45621)
0926 20:49:12.377422 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:12.377424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:12.377630 (+ 206us) server_negotiation.cc:407] Connection header received
0926 20:49:12.378539 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:12.378541 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:12.378551 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:12.378600 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:12.379197 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:12.379786 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:12.380806 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:12.380997 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:12.381060 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:12.383519 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:12.383521 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:12.383521 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:12.383528 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:12.385159 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:12.385575 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:12.385576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:12.385577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:12.385633 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:12.385889 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:12.385889 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:12.385890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:12.386007 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:12.386083 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:12.386352 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 20:49:12.386430 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 20:49:12.386716 6299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:12.377340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45621 (local address 127.0.98.2:52107)
0926 20:49:12.377508 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:12.377543 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:12.378447 (+ 904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:12.378662 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:12.378665 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:12.378922 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:12.379111 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:12.379114 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:12.379897 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:12.379898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:12.380696 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:12.380699 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:12.381076 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:12.381077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:12.381172 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:12.381829 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:12.381840 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:12.383422 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:12.385249 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:12.385256 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:12.385264 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:12.385483 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:12.385706 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:12.385707 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:12.385708 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:12.385819 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:12.386073 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:12.386078 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:12.386217 (+ 139us) client_negotiation.cc:769] Sending connection context
0926 20:49:12.386331 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 20:49:12.386486 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":90,"threads_started":1}
I20260926 20:49:12.387243 6297 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45621
I20260926 20:49:12.387357 6297 heartbeater.cc:464] Registering TS with master...
I20260926 20:49:12.387573 6297 heartbeater.cc:511] Master 127.0.98.62:45621 requested a full tablet report, sending...
I20260926 20:49:12.388012 5972 ts_manager.cc:194] Registered new tserver with Master: 5d69aba77175443bbcf9cda9c8e6dd04 (127.0.98.2:33109)
I20260926 20:49:12.388399 5972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52107
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 20:49:12.406865 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45621
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:49:12.487968 6302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:12.488173 6302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:12.488194 6302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:12.489274 6302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:12.489306 6302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:12.489331 6302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:49:12.490695 6302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:12.490869 6302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:12.491084 6302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:12.496022 6308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:12.495993 6310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:12.495993 6307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:12.496083 6302 server_base.cc:1034] running on GCE node
I20260926 20:49:12.496373 6302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:12.496564 6302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:12.497704 6302 hybrid_clock.cc:648] HybridClock initialized: now 1790455752497692 us; error 36 us; skew 500 ppm
Sep 26 20:49:12 dist-test-slave-hwlj krb5kdc[5927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:49:12.500064 6302 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 20:49:12.500495 6302 webserver.cc:460] Webserver started at http://127.0.98.3:42537/ using document root <none> and password file <none>
I20260926 20:49:12.500721 6302 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:12.500793 6302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:12.500947 6302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:49:12.501996 6302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/ts-2/data/instance:
uuid: "f47cbce9da674c6ca90319c6f0a6edce"
format_stamp: "Formatted at 2026-09-26 20:49:12 on dist-test-slave-hwlj"
server_key: "0a04b98637d9a9bf52f20a928f834f30"
server_key_iv: "9f56ab7554bea01b150f321c9c743e51"
server_key_version: "encryptionkey@0"
I20260926 20:49:12.502363 6302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f47cbce9da674c6ca90319c6f0a6edce"
format_stamp: "Formatted at 2026-09-26 20:49:12 on dist-test-slave-hwlj"
server_key: "0a04b98637d9a9bf52f20a928f834f30"
server_key_iv: "9f56ab7554bea01b150f321c9c743e51"
server_key_version: "encryptionkey@0"
I20260926 20:49:12.503688 6302 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 20:49:12.504451 6317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:12.504647 6302 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:49:12.504745 6302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/ts-2/wal
uuid: "f47cbce9da674c6ca90319c6f0a6edce"
format_stamp: "Formatted at 2026-09-26 20:49:12 on dist-test-slave-hwlj"
server_key: "0a04b98637d9a9bf52f20a928f834f30"
server_key_iv: "9f56ab7554bea01b150f321c9c743e51"
server_key_version: "encryptionkey@0"
I20260926 20:49:12.504827 6302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:12.517552 6302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:12.519541 6302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:12.519696 6302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:12.520031 6302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:49:12.520344 6302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:49:12.520397 6302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:12.520439 6302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:49:12.520467 6302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:12.526259 6302 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40455
I20260926 20:49:12.526295 6430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40455 every 8 connection(s)
I20260926 20:49:12.526610 6302 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
I20260926 20:49:12.531867 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 6302
I20260926 20:49:12.531999 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/minicluster-data/ts-2/wal/instance
I20260926 20:49:12.532227 392 external_mini_cluster.cc:1428] Setting key 202e93ac1df3839578d820b8a5a9651a
Sep 26 20:49:12 dist-test-slave-hwlj krb5kdc[5927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:49:12.536198 6012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:12.527326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59561 (local address 127.0.98.62:45621)
0926 20:49:12.527390 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:12.527391 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:12.527588 (+ 197us) server_negotiation.cc:407] Connection header received
0926 20:49:12.528648 (+ 1060us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:12.528649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:12.528662 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:12.528713 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:12.529335 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:12.529842 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:12.530843 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:12.531052 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:12.531135 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:12.533656 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:12.533657 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:12.533657 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:12.533664 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:12.534917 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:12.535311 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:12.535312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:12.535312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:12.535360 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:12.535578 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:12.535578 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:12.535579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:12.535676 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:12.535723 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:12.536069 (+ 346us) server_negotiation.cc:299] Negotiation successful
0926 20:49:12.536134 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 20:49:12.536357 6433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:12.527303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45621 (local address 127.0.98.3:59561)
0926 20:49:12.527477 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:12.527511 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:12.528530 (+ 1019us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:12.528781 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:12.528785 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:12.529039 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:12.529223 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:12.529227 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:12.529933 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:12.529935 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:12.530734 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:12.530737 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:12.531143 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:12.531145 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:12.531274 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:12.531912 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:12.531920 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:12.533562 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:12.535004 (+ 1442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:12.535010 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:12.535012 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:12.535243 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:12.535420 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:12.535420 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:12.535421 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:12.535525 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:12.535734 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:12.535739 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:12.535949 (+ 210us) client_negotiation.cc:769] Sending connection context
0926 20:49:12.536039 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 20:49:12.536162 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":84,"threads_started":1}
I20260926 20:49:12.536880 6431 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45621
I20260926 20:49:12.536986 6431 heartbeater.cc:464] Registering TS with master...
I20260926 20:49:12.537189 6431 heartbeater.cc:511] Master 127.0.98.62:45621 requested a full tablet report, sending...
I20260926 20:49:12.537544 5972 ts_manager.cc:194] Registered new tserver with Master: f47cbce9da674c6ca90319c6f0a6edce (127.0.98.3:40455)
I20260926 20:49:12.537925 5972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59561
I20260926 20:49:12.543988 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/krb5kdc/test-admin.keytab.
Sep 26 20:49:12 dist-test-slave-hwlj krb5kdc[5927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:49:12.560992 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 20:49:12.562660 6438 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:12.561864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33712 (local address 127.0.98.1:43253)
0926 20:49:12.562040 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:12.562045 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:12.562055 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:49:12.562088 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:12.562090 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:12.562104 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:12.562176 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:12.562579 (+ 403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33712: BlockingRecv error: recv got EOF from 127.0.0.1:33712 (error 108)
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":101,"threads_started":1}
W20260926 20:49:12.562796 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/krb5kdc/krb5cc' of type FILE
Sep 26 20:49:12 dist-test-slave-hwlj krb5kdc[5927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:49:12.563850 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 20:49:12 dist-test-slave-hwlj krb5kdc[5927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:49:12.564781 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 20:49:12 dist-test-slave-hwlj krb5kdc[5927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 20:49:12.572371 6438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:12.565361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33722 (local address 127.0.98.1:43253)
0926 20:49:12.565412 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:12.565413 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:12.565475 (+ 62us) server_negotiation.cc:407] Connection header received
0926 20:49:12.565533 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:12.565534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:12.565562 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:12.565615 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:12.565994 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:12.566706 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:12.567422 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:12.567735 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:12.567860 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:12.569850 (+ 1990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:12.569857 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:12.569857 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:12.569864 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:12.571267 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:12.571576 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:12.571579 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:12.571579 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:12.571629 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:12.571913 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:12.571914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:12.571914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:12.572037 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:12.572140 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:12.572224 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 20:49:12.572294 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
W20260926 20:49:12.574301 6438 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:12.573697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33732 (local address 127.0.98.1:43253)
0926 20:49:12.573752 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:12.573754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:12.573803 (+ 49us) server_negotiation.cc:407] Connection header received
0926 20:49:12.573851 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:12.573852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:12.573860 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:12.573905 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:12.574269 (+ 364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33732: BlockingRecv error: recv got EOF from 127.0.0.1:33732 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260926 20:49:12.574405 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/krb5kdc/krb5cc' of type FILE
Sep 26 20:49:12 dist-test-slave-hwlj krb5kdc[5927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:49:12.575400 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 20:49:12 dist-test-slave-hwlj krb5kdc[5927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:49:12.576309 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 20:49:12 dist-test-slave-hwlj krb5kdc[5927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 20:49:12.583339 6438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:12.576830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33736 (local address 127.0.98.1:43253)
0926 20:49:12.576880 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:12.576883 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:12.576895 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:49:12.576912 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:12.576912 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:12.576920 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:12.576964 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:12.577334 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:12.577839 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:12.578823 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:12.579013 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:12.579067 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:12.581215 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:12.581216 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:12.581216 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:12.581222 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:12.582402 (+ 1180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:12.582737 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:12.582738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:12.582739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:12.582798 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:12.582957 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:12.582958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:12.582958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:12.583041 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:12.583107 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:12.583221 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 20:49:12.583281 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260926 20:49:12.585093 6438 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:12.584582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33752 (local address 127.0.98.1:43253)
0926 20:49:12.584648 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:12.584650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:12.584679 (+ 29us) server_negotiation.cc:407] Connection header received
0926 20:49:12.584696 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:12.584697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:12.584705 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:12.584753 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:12.585037 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33752: BlockingRecv error: recv got EOF from 127.0.0.1:33752 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260926 20:49:12.585170 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455725523895-392-0/krb5kdc/krb5cc' of type FILE
Sep 26 20:49:12 dist-test-slave-hwlj krb5kdc[5927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:49:12.586177 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 20:49:12 dist-test-slave-hwlj krb5kdc[5927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:49:12.586985 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 20:49:12 dist-test-slave-hwlj krb5kdc[5927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 20:49:12.596027 6438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:12.587579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33758 (local address 127.0.98.1:43253)
0926 20:49:12.587665 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:12.587666 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:12.587705 (+ 39us) server_negotiation.cc:407] Connection header received
0926 20:49:12.587722 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:12.587723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:12.587730 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:12.587862 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:12.588196 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:12.589001 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:12.589749 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:12.590093 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:12.590170 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:12.592903 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:12.592905 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:12.592905 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:12.592913 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:12.594519 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:12.594966 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:12.594967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:12.594967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:12.595037 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:12.595338 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:12.595340 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:12.595340 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:12.595482 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:12.595595 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:12.595776 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 20:49:12.595865 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 20:49:12.596803 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 6034
I20260926 20:49:12.601229 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 6168
I20260926 20:49:12.605751 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 6302
I20260926 20:49:12.610128 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 5940
2026-09-26T20:49:12Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1840 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:49:12 dist-test-slave-hwlj krb5kdc[6451](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:49:12 dist-test-slave-hwlj krb5kdc[6451](info): set up 2 sockets
Sep 26 20:49:12 dist-test-slave-hwlj krb5kdc[6451](info): commencing operation
krb5kdc: starting...
W20260926 20:49:13.771979 392 mini_kdc.cc:121] Time spent starting KDC: real 1.141s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/test-user.keytab.
Sep 26 20:49:13 dist-test-slave-hwlj krb5kdc[6451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:49:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:49:13Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260926 20:49:13.891117 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40969
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40969
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:49:13.970762 6464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:13.971016 6464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:13.971056 6464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:13.972268 6464 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:49:13.972337 6464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:13.972361 6464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:13.972383 6464 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:49:13.972414 6464 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:49:13.973865 6464 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:13.974148 6464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:13.974380 6464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:13.979691 6469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:13.979733 6472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:13.979691 6464 server_base.cc:1034] running on GCE node
W20260926 20:49:13.979974 6470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:13.980260 6464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:13.980515 6464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:13.981652 6464 hybrid_clock.cc:648] HybridClock initialized: now 1790455753981638 us; error 30 us; skew 500 ppm
Sep 26 20:49:13 dist-test-slave-hwlj krb5kdc[6451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455753, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:49:13.987479 6464 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260926 20:49:13.988016 6464 webserver.cc:460] Webserver started at http://127.0.98.62:41301/ using document root <none> and password file <none>
I20260926 20:49:13.988248 6464 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:13.988296 6464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:13.988447 6464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:49:13.989554 6464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-0/data/instance:
uuid: "4e93f52007174325a60947bfdcddc4d0"
format_stamp: "Formatted at 2026-09-26 20:49:13 on dist-test-slave-hwlj"
server_key: "a6455ccdd00507a62a08594777d20941"
server_key_iv: "44a571df2db36b0da0da27d918423845"
server_key_version: "encryptionkey@0"
I20260926 20:49:13.989934 6464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-0/wal/instance:
uuid: "4e93f52007174325a60947bfdcddc4d0"
format_stamp: "Formatted at 2026-09-26 20:49:13 on dist-test-slave-hwlj"
server_key: "a6455ccdd00507a62a08594777d20941"
server_key_iv: "44a571df2db36b0da0da27d918423845"
server_key_version: "encryptionkey@0"
I20260926 20:49:13.991310 6464 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 20:49:13.992188 6479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:13.992395 6464 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 20:49:13.992507 6464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "4e93f52007174325a60947bfdcddc4d0"
format_stamp: "Formatted at 2026-09-26 20:49:13 on dist-test-slave-hwlj"
server_key: "a6455ccdd00507a62a08594777d20941"
server_key_iv: "44a571df2db36b0da0da27d918423845"
server_key_version: "encryptionkey@0"
I20260926 20:49:13.992585 6464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:14.020428 6464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:14.022497 6464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:14.022675 6464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:14.026855 6464 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40969
I20260926 20:49:14.026893 6531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40969 every 8 connection(s)
I20260926 20:49:14.027266 6464 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:49:14.028324 6532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:49:14.030610 6532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0: Bootstrap starting.
I20260926 20:49:14.031121 6532 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0: Neither blocks nor log segments found. Creating new log.
I20260926 20:49:14.031411 6532 log.cc:824] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0: Log is configured to *not* fsync() on all Append() calls
I20260926 20:49:14.032083 6532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0: No bootstrap required, opened a new log
I20260926 20:49:14.033629 6532 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } }
I20260926 20:49:14.033768 6532 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:49:14.033790 6532 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e93f52007174325a60947bfdcddc4d0, State: Initialized, Role: FOLLOWER
I20260926 20:49:14.033927 6532 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } }
I20260926 20:49:14.034010 6532 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:49:14.034032 6532 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:49:14.034060 6532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:49:14.034830 6532 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } }
I20260926 20:49:14.034916 6532 leader_election.cc:302] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e93f52007174325a60947bfdcddc4d0; no voters:
I20260926 20:49:14.035081 6532 leader_election.cc:288] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:49:14.035116 6535 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:49:14.035235 6535 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0 [term 1 LEADER]: Becoming Leader. State: Replica: 4e93f52007174325a60947bfdcddc4d0, State: Running, Role: LEADER
I20260926 20:49:14.035315 6535 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } }
I20260926 20:49:14.035349 6532 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:49:14.035676 6537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e93f52007174325a60947bfdcddc4d0. Latest consensus state: current_term: 1 leader_uuid: "4e93f52007174325a60947bfdcddc4d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } }
I20260926 20:49:14.035712 6536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e93f52007174325a60947bfdcddc4d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } }
I20260926 20:49:14.035778 6537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0 [sys.catalog]: This master's current role is: LEADER
I20260926 20:49:14.035815 6536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0 [sys.catalog]: This master's current role is: LEADER
I20260926 20:49:14.036288 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 6464
I20260926 20:49:14.036414 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-0/wal/instance
I20260926 20:49:14.036485 6541 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:49:14.036655 392 external_mini_cluster.cc:1428] Setting key 8c6f76e7fa2f2d8c0022736d5df8236b
I20260926 20:49:14.036981 6541 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:49:14.038538 6541 catalog_manager.cc:1347] Generated new cluster ID: d234dc66a6104be1b78c57501e0b0d6d
I20260926 20:49:14.038590 6541 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 20:49:14 dist-test-slave-hwlj krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 20:49:14.048532 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:14.037153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43476 (local address 127.0.98.62:40969)
0926 20:49:14.037316 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:14.037324 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:14.037581 (+ 257us) server_negotiation.cc:407] Connection header received
0926 20:49:14.037803 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:14.037811 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:14.037887 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:14.038146 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:14.038884 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:14.040308 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:14.041382 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:14.041984 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:14.042075 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:14.044874 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:14.044879 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:14.044880 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:14.044893 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:14.046911 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:14.047277 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:14.047281 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:14.047281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:14.047400 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:14.047598 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:14.047599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:14.047599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:14.047934 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:14.048104 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:14.048173 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 20:49:14.048306 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":75,"threads_started":1}
I20260926 20:49:14.051172 6541 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:49:14.051868 6541 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:49:14.059217 6541 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0: Generated new TSK 0
I20260926 20:49:14.059382 6541 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260926 20:49:14.091493 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40969
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:49:14.175665 6558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:14.176008 6558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:14.176054 6558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:14.177425 6558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:14.177513 6558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:14.177575 6558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:49:14.179303 6558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:14.179615 6558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:14.179952 6558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:14.185357 6564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:14.185377 6563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:14.185395 6566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:14.185601 6558 server_base.cc:1034] running on GCE node
I20260926 20:49:14.185866 6558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:14.186101 6558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:14.187256 6558 hybrid_clock.cc:648] HybridClock initialized: now 1790455754187239 us; error 48 us; skew 500 ppm
Sep 26 20:49:14 dist-test-slave-hwlj krb5kdc[6451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455754, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:49:14.189813 6558 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260926 20:49:14.190263 6558 webserver.cc:460] Webserver started at http://127.0.98.1:33941/ using document root <none> and password file <none>
I20260926 20:49:14.190493 6558 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:14.190537 6558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:14.190680 6558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:49:14.191704 6558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-0/data/instance:
uuid: "e728cd363cea4949a2a5d07380f215d8"
format_stamp: "Formatted at 2026-09-26 20:49:14 on dist-test-slave-hwlj"
server_key: "b3d6bddd0112895917eb6c1122194e20"
server_key_iv: "26ddcba06655abf785ef003e648f7999"
server_key_version: "encryptionkey@0"
I20260926 20:49:14.192122 6558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e728cd363cea4949a2a5d07380f215d8"
format_stamp: "Formatted at 2026-09-26 20:49:14 on dist-test-slave-hwlj"
server_key: "b3d6bddd0112895917eb6c1122194e20"
server_key_iv: "26ddcba06655abf785ef003e648f7999"
server_key_version: "encryptionkey@0"
I20260926 20:49:14.193426 6558 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:49:14.194223 6573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:14.194415 6558 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:49:14.194497 6558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-0/wal
uuid: "e728cd363cea4949a2a5d07380f215d8"
format_stamp: "Formatted at 2026-09-26 20:49:14 on dist-test-slave-hwlj"
server_key: "b3d6bddd0112895917eb6c1122194e20"
server_key_iv: "26ddcba06655abf785ef003e648f7999"
server_key_version: "encryptionkey@0"
I20260926 20:49:14.194551 6558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:14.205586 6558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:14.207720 6558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:14.207923 6558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:14.208138 6558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:49:14.208403 6558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:49:14.208432 6558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:14.208454 6558 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:49:14.208467 6558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:14.214178 6558 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46533
I20260926 20:49:14.214283 6686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46533 every 8 connection(s)
I20260926 20:49:14.214604 6558 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:49:14.216346 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 6558
I20260926 20:49:14.216450 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-0/wal/instance
I20260926 20:49:14.216666 392 external_mini_cluster.cc:1428] Setting key 99fc97f72b38a3733dc1463b0833640a
Sep 26 20:49:14 dist-test-slave-hwlj krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455754, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 20:49:14.225689 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:14.215342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56617 (local address 127.0.98.62:40969)
0926 20:49:14.215390 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:14.215401 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:14.215627 (+ 226us) server_negotiation.cc:407] Connection header received
0926 20:49:14.216193 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:14.216194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:14.216211 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:14.216295 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:14.217600 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:14.218410 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:14.219853 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:14.220097 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:14.220170 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:14.222699 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:14.222701 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:14.222701 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:14.222709 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:14.224225 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:14.224647 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:14.224648 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:14.224649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:14.224708 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:14.224989 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:14.224991 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:14.224991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:14.225152 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:14.225243 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:14.225551 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 20:49:14.225623 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 20:49:14.225975 6689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:14.215333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40969 (local address 127.0.98.1:56617)
0926 20:49:14.215501 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:14.215538 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:14.216044 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:14.216349 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:14.216352 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:14.217167 (+ 815us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:14.217457 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:14.217463 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:14.218503 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:14.218503 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:14.219772 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:14.219775 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:14.220204 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:14.220206 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:14.220355 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:14.220896 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:14.220907 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:14.222562 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:14.224334 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:14.224342 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:14.224344 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:14.224566 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:14.224782 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:14.224782 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:14.224783 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:14.224894 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:14.225265 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:14.225271 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:14.225404 (+ 133us) client_negotiation.cc:769] Sending connection context
0926 20:49:14.225521 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 20:49:14.225671 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":92,"threads_started":1}
I20260926 20:49:14.226593 6687 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40969
I20260926 20:49:14.226712 6687 heartbeater.cc:464] Registering TS with master...
I20260926 20:49:14.226926 6687 heartbeater.cc:511] Master 127.0.98.62:40969 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260926 20:49:14.227541 6496 ts_manager.cc:194] Registered new tserver with Master: e728cd363cea4949a2a5d07380f215d8 (127.0.98.1:46533)
I20260926 20:49:14.228617 6496 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56617
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260926 20:49:14.243853 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40969
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:49:14.323565 6692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:14.323815 6692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:14.323856 6692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:14.325006 6692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:14.325070 6692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:14.325114 6692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:49:14.326576 6692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:14.326841 6692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:14.327064 6692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:14.332091 6700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:14.332109 6697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:14.332113 6698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:14.332237 6692 server_base.cc:1034] running on GCE node
I20260926 20:49:14.332582 6692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:14.332823 6692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:14.333987 6692 hybrid_clock.cc:648] HybridClock initialized: now 1790455754333973 us; error 37 us; skew 500 ppm
Sep 26 20:49:14 dist-test-slave-hwlj krb5kdc[6451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455754, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:49:14.336740 6692 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260926 20:49:14.337225 6692 webserver.cc:460] Webserver started at http://127.0.98.2:43457/ using document root <none> and password file <none>
I20260926 20:49:14.337460 6692 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:14.337527 6692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:14.337658 6692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:49:14.338671 6692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-1/data/instance:
uuid: "01cba43c9838497f8d037dec2c16e44a"
format_stamp: "Formatted at 2026-09-26 20:49:14 on dist-test-slave-hwlj"
server_key: "778f4dd9cd7606168935f47996d6d73f"
server_key_iv: "8466fedcfecf4a68ca6c2aeb966e6d75"
server_key_version: "encryptionkey@0"
I20260926 20:49:14.339030 6692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-1/wal/instance:
uuid: "01cba43c9838497f8d037dec2c16e44a"
format_stamp: "Formatted at 2026-09-26 20:49:14 on dist-test-slave-hwlj"
server_key: "778f4dd9cd7606168935f47996d6d73f"
server_key_iv: "8466fedcfecf4a68ca6c2aeb966e6d75"
server_key_version: "encryptionkey@0"
I20260926 20:49:14.340422 6692 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 20:49:14.341123 6707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:14.341307 6692 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:49:14.341401 6692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-1/wal
uuid: "01cba43c9838497f8d037dec2c16e44a"
format_stamp: "Formatted at 2026-09-26 20:49:14 on dist-test-slave-hwlj"
server_key: "778f4dd9cd7606168935f47996d6d73f"
server_key_iv: "8466fedcfecf4a68ca6c2aeb966e6d75"
server_key_version: "encryptionkey@0"
I20260926 20:49:14.341482 6692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:14.364456 6692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:14.366508 6692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:14.366673 6692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:14.366901 6692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:49:14.367215 6692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:49:14.367269 6692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:14.367304 6692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:49:14.367332 6692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:14.373028 6692 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44443
I20260926 20:49:14.373080 6820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44443 every 8 connection(s)
I20260926 20:49:14.373399 6692 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
I20260926 20:49:14.378624 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 6692
I20260926 20:49:14.378707 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-1/wal/instance
I20260926 20:49:14.378906 392 external_mini_cluster.cc:1428] Setting key 5da567f3e75c2c3ca31fde53bcfcfd15
Sep 26 20:49:14 dist-test-slave-hwlj krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455754, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 20:49:14.383054 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:14.374124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46019 (local address 127.0.98.62:40969)
0926 20:49:14.374186 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:14.374188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:14.374369 (+ 181us) server_negotiation.cc:407] Connection header received
0926 20:49:14.374848 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:14.374849 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:14.374858 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:14.374916 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:14.375865 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:14.376337 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:14.377391 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:14.377599 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:14.377698 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:14.380129 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:14.380131 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:14.380131 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:14.380139 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:14.381708 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:14.382112 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:14.382113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:14.382113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:14.382170 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:14.382426 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:14.382427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:14.382427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:14.382538 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:14.382613 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:14.382887 (+ 274us) server_negotiation.cc:299] Negotiation successful
0926 20:49:14.382967 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 20:49:14.383262 6823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:14.374072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40969 (local address 127.0.98.2:46019)
0926 20:49:14.374237 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:14.374280 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:14.374761 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:14.374970 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:14.374973 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:14.375537 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:14.375726 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:14.375730 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:14.376416 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:14.376417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:14.377277 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:14.377279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:14.377709 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:14.377709 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:14.377815 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:14.378359 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:14.378368 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:14.380038 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:14.381803 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:14.381809 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:14.381811 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:14.382025 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:14.382236 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:14.382237 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:14.382237 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:14.382355 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:14.382604 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:14.382610 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:14.382753 (+ 143us) client_negotiation.cc:769] Sending connection context
0926 20:49:14.382856 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 20:49:14.383008 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":96,"threads_started":1}
I20260926 20:49:14.384194 6821 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40969
I20260926 20:49:14.384276 6821 heartbeater.cc:464] Registering TS with master...
I20260926 20:49:14.384467 6821 heartbeater.cc:511] Master 127.0.98.62:40969 requested a full tablet report, sending...
I20260926 20:49:14.384852 6496 ts_manager.cc:194] Registered new tserver with Master: 01cba43c9838497f8d037dec2c16e44a (127.0.98.2:44443)
I20260926 20:49:14.385378 6496 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46019
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260926 20:49:14.404320 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40969
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:49:14.485387 6826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:14.485618 6826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:14.485643 6826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:14.486758 6826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:14.486796 6826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:14.486822 6826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:49:14.488348 6826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:14.488539 6826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:14.488718 6826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:49:14.493731 6826 server_base.cc:1034] running on GCE node
W20260926 20:49:14.493713 6832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:14.493714 6831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:14.493743 6834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:14.494133 6826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:14.494413 6826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:14.495587 6826 hybrid_clock.cc:648] HybridClock initialized: now 1790455754495574 us; error 25 us; skew 500 ppm
Sep 26 20:49:14 dist-test-slave-hwlj krb5kdc[6451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455754, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:49:14.498358 6826 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260926 20:49:14.498888 6826 webserver.cc:460] Webserver started at http://127.0.98.3:42093/ using document root <none> and password file <none>
I20260926 20:49:14.499120 6826 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:14.499172 6826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:14.499338 6826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:49:14.500483 6826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-2/data/instance:
uuid: "345595a322944cad8c9ac66f823413a7"
format_stamp: "Formatted at 2026-09-26 20:49:14 on dist-test-slave-hwlj"
server_key: "78304a518f37b53d9e3f84f7399d2efa"
server_key_iv: "74bd3ba1e2a9aaa740cd4549dca69391"
server_key_version: "encryptionkey@0"
I20260926 20:49:14.500849 6826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-2/wal/instance:
uuid: "345595a322944cad8c9ac66f823413a7"
format_stamp: "Formatted at 2026-09-26 20:49:14 on dist-test-slave-hwlj"
server_key: "78304a518f37b53d9e3f84f7399d2efa"
server_key_iv: "74bd3ba1e2a9aaa740cd4549dca69391"
server_key_version: "encryptionkey@0"
I20260926 20:49:14.502177 6826 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 20:49:14.502875 6841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:14.503067 6826 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:14.503173 6826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-2/wal
uuid: "345595a322944cad8c9ac66f823413a7"
format_stamp: "Formatted at 2026-09-26 20:49:14 on dist-test-slave-hwlj"
server_key: "78304a518f37b53d9e3f84f7399d2efa"
server_key_iv: "74bd3ba1e2a9aaa740cd4549dca69391"
server_key_version: "encryptionkey@0"
I20260926 20:49:14.503245 6826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:14.514235 6826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:14.516593 6826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:14.516760 6826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:14.516980 6826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:49:14.517295 6826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:49:14.517333 6826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:14.517354 6826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:49:14.517405 6826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:14.523010 6826 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39349
I20260926 20:49:14.523064 6954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39349 every 8 connection(s)
I20260926 20:49:14.523367 6826 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
I20260926 20:49:14.528955 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 6826
I20260926 20:49:14.529067 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-2/wal/instance
Sep 26 20:49:14 dist-test-slave-hwlj krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455754, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 20:49:14.529273 392 external_mini_cluster.cc:1428] Setting key 521a607ba51d9f17b415aedd13b704d0
I20260926 20:49:14.532437 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:14.524099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52259 (local address 127.0.98.62:40969)
0926 20:49:14.524144 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:14.524147 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:14.524336 (+ 189us) server_negotiation.cc:407] Connection header received
0926 20:49:14.524849 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:14.524850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:14.524864 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:14.524927 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:14.525859 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:14.526357 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:14.527337 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:14.527546 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:14.527634 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:14.529858 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:14.529859 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:14.529859 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:14.529868 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:14.531134 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:14.531486 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:14.531486 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:14.531487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:14.531537 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:14.531739 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:14.531740 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:14.531740 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:14.531896 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:14.531971 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:14.532272 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 20:49:14.532349 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 20:49:14.532625 6957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:14.524060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40969 (local address 127.0.98.3:52259)
0926 20:49:14.524208 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:14.524249 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:14.524747 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:14.524995 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:14.524999 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:14.525576 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:14.525760 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:14.525764 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:14.526447 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:14.526448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:14.527246 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:14.527248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:14.527645 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:14.527646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:14.527790 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:14.528231 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:14.528238 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:14.529740 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:14.531211 (+ 1471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:14.531216 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:14.531217 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:14.531422 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:14.531593 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:14.531594 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:14.531594 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:14.531684 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:14.531978 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:14.531984 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:14.532117 (+ 133us) client_negotiation.cc:769] Sending connection context
0926 20:49:14.532238 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 20:49:14.532395 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":63,"threads_started":1}
I20260926 20:49:14.533064 6955 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40969
I20260926 20:49:14.533167 6955 heartbeater.cc:464] Registering TS with master...
I20260926 20:49:14.533335 6955 heartbeater.cc:511] Master 127.0.98.62:40969 requested a full tablet report, sending...
I20260926 20:49:14.533699 6496 ts_manager.cc:194] Registered new tserver with Master: 345595a322944cad8c9ac66f823413a7 (127.0.98.3:39349)
I20260926 20:49:14.534067 6496 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52259
I20260926 20:49:14.540863 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260926 20:49:14.564641 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:44089
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40969,127.0.98.61:44089
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:49:14.644330 6960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:14.644586 6960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:14.644608 6960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:14.645731 6960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:49:14.645768 6960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:14.645782 6960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:14.645794 6960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:49:14.645807 6960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:49:14.647183 6960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40969,127.0.98.61:44089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:44089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
W20260926 20:49:14.647279 6960 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:40969,127.0.98.61:44089'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 20:49:14.647387 6960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:14.647554 6960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:14.652487 6965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:14.652468 6966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:14.652565 6960 server_base.cc:1034] running on GCE node
W20260926 20:49:14.652468 6968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:14.652845 6960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:14.653100 6960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:14.654247 6960 hybrid_clock.cc:648] HybridClock initialized: now 1790455754654237 us; error 29 us; skew 500 ppm
Sep 26 20:49:14 dist-test-slave-hwlj krb5kdc[6451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455754, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:49:14.656751 6960 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260926 20:49:14.657135 6960 webserver.cc:460] Webserver started at http://127.0.98.61:42865/ using document root <none> and password file <none>
I20260926 20:49:14.657325 6960 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:14.657364 6960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:14.657449 6960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:49:14.658510 6960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-1/data/instance:
uuid: "27e19353428e42f09a3f45b96fb8a573"
format_stamp: "Formatted at 2026-09-26 20:49:14 on dist-test-slave-hwlj"
server_key: "5b196f2d96fb00870cf7f1d89d81144a"
server_key_iv: "7d88f0d4e486e79dc84256a5d46a3129"
server_key_version: "encryptionkey@0"
I20260926 20:49:14.658844 6960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-1/wal/instance:
uuid: "27e19353428e42f09a3f45b96fb8a573"
format_stamp: "Formatted at 2026-09-26 20:49:14 on dist-test-slave-hwlj"
server_key: "5b196f2d96fb00870cf7f1d89d81144a"
server_key_iv: "7d88f0d4e486e79dc84256a5d46a3129"
server_key_version: "encryptionkey@0"
I20260926 20:49:14.660174 6960 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:49:14.660933 6975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:14.661095 6960 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:49:14.661178 6960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-1/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-1/wal
uuid: "27e19353428e42f09a3f45b96fb8a573"
format_stamp: "Formatted at 2026-09-26 20:49:14 on dist-test-slave-hwlj"
server_key: "5b196f2d96fb00870cf7f1d89d81144a"
server_key_iv: "7d88f0d4e486e79dc84256a5d46a3129"
server_key_version: "encryptionkey@0"
I20260926 20:49:14.661231 6960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:14.669060 6960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:14.671227 6960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:14.671336 6960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:14.675055 6960 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:44089
I20260926 20:49:14.675130 7027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:44089 every 8 connection(s)
I20260926 20:49:14.675455 6960 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-1/data/info.pb
I20260926 20:49:14.676472 7028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:49:14.678076 7028 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:49:14.678931 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 6960
Sep 26 20:49:14 dist-test-slave-hwlj krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455754, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 26 20:49:14 dist-test-slave-hwlj krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 20:49:14.688241 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:14.678848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43480 (local address 127.0.98.62:40969)
0926 20:49:14.678901 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:14.678923 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:14.679123 (+ 200us) server_negotiation.cc:407] Connection header received
0926 20:49:14.679891 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:14.679892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:14.679908 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:14.679996 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:14.680556 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:14.681055 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:14.681973 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:14.682167 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:14.682242 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:14.684710 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:14.684712 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:14.684712 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:14.684721 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:14.686486 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:14.686961 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:14.686963 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:14.686963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:14.687025 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:14.687314 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:14.687315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:14.687315 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:14.687528 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:14.687592 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:14.687997 (+ 405us) server_negotiation.cc:299] Negotiation successful
0926 20:49:14.688101 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 20:49:14.688582 7030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:14.678804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40969 (local address 127.0.0.1:43480)
0926 20:49:14.678993 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:14.679042 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:14.679794 (+ 752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:14.680022 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:14.680025 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:14.680300 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:14.680464 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:14.680468 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:14.681158 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:14.681159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:14.681872 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:14.681874 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:14.682260 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:14.682261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:14.682381 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:14.682924 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:14.682934 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:14.684568 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:14.686597 (+ 2029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:14.686603 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:14.686605 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:14.686869 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:14.687109 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:14.687110 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:14.687110 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:14.687240 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:14.687617 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:14.687622 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:14.687809 (+ 187us) client_negotiation.cc:769] Sending connection context
0926 20:49:14.687946 (+ 137us) client_negotiation.cc:240] Negotiation successful
0926 20:49:14.688102 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":91,"threads_started":1}
I20260926 20:49:14.688758 7032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:14.679449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37072 (local address 127.0.98.61:44089)
0926 20:49:14.679586 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:14.679590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:14.679616 (+ 26us) server_negotiation.cc:407] Connection header received
0926 20:49:14.679787 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:14.679798 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:14.679822 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:14.679922 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:14.680531 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:14.681315 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:14.682048 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:14.682353 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:14.682442 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:14.685204 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:14.685228 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:14.685228 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:14.685236 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:14.686985 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:14.687402 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:14.687405 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:14.687405 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:14.687456 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:14.687699 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:14.687701 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:14.687702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:14.687929 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:14.688125 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:14.688564 (+ 439us) server_negotiation.cc:299] Negotiation successful
0926 20:49:14.688642 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":69,"threads_started":1}
I20260926 20:49:14.689317 7028 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44089 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:49:14 dist-test-slave-hwlj krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455754, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 20:49:14.695585 7030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:14.689610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44089 (local address 127.0.0.1:37074)
0926 20:49:14.689667 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:14.689678 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:14.689733 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:14.689852 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:14.689852 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:14.689983 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:14.690025 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:14.690026 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:14.690646 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:14.690647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:14.691155 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:14.691156 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:14.691537 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:14.691538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:14.691608 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:14.692048 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:14.692053 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:14.693154 (+ 1101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:14.694547 (+ 1393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:14.694548 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:14.694548 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:14.694706 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:14.694944 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:14.694945 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:14.694945 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:14.694999 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:14.695261 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:14.695261 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:14.695301 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 20:49:14.695379 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 20:49:14.695474 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 20:49:14.695595 7032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:14.689641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37074 (local address 127.0.98.61:44089)
0926 20:49:14.689670 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:14.689672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:14.689718 (+ 46us) server_negotiation.cc:407] Connection header received
0926 20:49:14.689785 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:14.689786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:14.689796 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:14.689849 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:14.690107 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:14.690556 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:14.691249 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:14.691450 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:14.691534 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:14.693251 (+ 1717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:14.693252 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:14.693253 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:14.693259 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:14.694446 (+ 1187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:14.694789 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:14.694790 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:14.694791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:14.694847 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:14.695072 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:14.695073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:14.695074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:14.695177 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:14.695269 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:14.695409 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 20:49:14.695523 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":17}
I20260926 20:49:14.696957 7028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573: Bootstrap starting.
I20260926 20:49:14.697863 7028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573: Neither blocks nor log segments found. Creating new log.
I20260926 20:49:14.698097 7028 log.cc:824] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573: Log is configured to *not* fsync() on all Append() calls
I20260926 20:49:14.698858 7028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573: No bootstrap required, opened a new log
I20260926 20:49:14.700542 7028 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44089 } }
I20260926 20:49:14.700673 7028 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:49:14.700697 7028 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27e19353428e42f09a3f45b96fb8a573, State: Initialized, Role: FOLLOWER
I20260926 20:49:14.700855 7028 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44089 } }
I20260926 20:49:14.701114 7034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44089 } } }
I20260926 20:49:14.701169 7028 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:49:14.701225 7034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 20:49:14.702872 7045 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:49:14.702944 7045 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:49:14.704717 6960 master_runner.cc:428] Detected that this master 27e19353428e42f09a3f45b96fb8a573 is joining an existing cluster
I20260926 20:49:14.704757 6960 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:44089
I20260926 20:49:14.712224 7032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:14.706249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37076 (local address 127.0.98.61:44089)
0926 20:49:14.706290 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:14.706292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:14.706300 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:49:14.706318 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:14.706319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:14.706331 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:14.706392 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:14.706731 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:14.707310 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:14.708095 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:14.708303 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:14.708372 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:14.709362 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:14.709363 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:14.709363 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:14.709369 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:14.710969 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:14.711365 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:14.711366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:14.711366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:14.711417 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:14.711611 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:14.711611 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:14.711612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:14.711740 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:14.711807 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:14.711994 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 20:49:14.712114 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":22}
I20260926 20:49:14.712225 7052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:14.706100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44089 (local address 127.0.0.1:37076)
0926 20:49:14.706209 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:14.706219 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:14.706262 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:14.706417 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:14.706418 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:14.706593 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:14.706655 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:14.706655 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:14.707384 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:14.707385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:14.708012 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:14.708014 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:14.708390 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:14.708391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:14.708456 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:14.708896 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:14.708901 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:14.709274 (+ 373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:14.711051 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:14.711052 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:14.711052 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:14.711279 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:14.711487 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:14.711488 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:14.711488 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:14.711543 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:14.711847 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:14.711848 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:14.711875 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 20:49:14.711960 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 20:49:14.712063 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":68,"threads_started":1}
I20260926 20:49:14.713006 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:14.705810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43484 (local address 127.0.98.62:40969)
0926 20:49:14.705924 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:14.705927 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:14.705968 (+ 41us) server_negotiation.cc:407] Connection header received
0926 20:49:14.706049 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:14.706049 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:14.706060 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:14.706108 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:14.706596 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:14.707359 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:14.708490 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:14.708792 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:14.708833 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:14.709889 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:14.709890 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:14.709890 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:14.709896 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:14.711608 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:14.712029 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:14.712030 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:14.712031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:14.712086 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:14.712496 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:14.712497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:14.712497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:14.712634 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:14.712678 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:14.712887 (+ 209us) server_negotiation.cc:299] Negotiation successful
0926 20:49:14.712948 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260926 20:49:14.713145 7051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:14.705781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40969 (local address 127.0.0.1:43484)
0926 20:49:14.705913 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:14.705925 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:14.705984 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:14.706110 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:14.706110 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:14.706405 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:14.706505 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:14.706507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:14.707455 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:14.707456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:14.708388 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:14.708390 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:14.708910 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:14.708911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:14.708972 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:14.709364 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:14.709369 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:14.709789 (+ 420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:14.711698 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:14.711699 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:14.711699 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:14.711943 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:14.712157 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:14.712157 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:14.712158 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:14.712229 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:14.712725 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:14.712726 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:14.712765 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 20:49:14.712845 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 20:49:14.712937 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":78,"threads_started":1}
Sep 26 20:49:14 dist-test-slave-hwlj krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455753, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 20:49:14.723152 7032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:14.715562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37086 (local address 127.0.98.61:44089)
0926 20:49:14.715601 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:14.715603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:14.715822 (+ 219us) server_negotiation.cc:407] Connection header received
0926 20:49:14.715859 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:14.715860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:14.715870 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:14.715931 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:14.716444 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:14.716939 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:14.718054 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:14.718236 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:14.718305 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:14.720540 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:14.720542 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:14.720542 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:14.720549 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:14.721918 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:14.722335 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:14.722336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:14.722337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:14.722398 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:14.722601 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:14.722602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:14.722602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:14.722725 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:14.722801 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:14.722980 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 20:49:14.723058 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 20:49:14.723181 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:14.715490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44089 (local address 127.0.0.1:37086)
0926 20:49:14.715634 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:14.715675 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:14.715781 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:14.715955 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:14.715958 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:14.716301 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:14.716359 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:14.716361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:14.717020 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:14.717021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:14.717963 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:14.717965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:14.718321 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:14.718322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:14.718405 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:14.718925 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:14.718930 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:14.720452 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:14.722026 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:14.722035 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:14.722037 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:14.722259 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:14.722467 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:14.722468 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:14.722468 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:14.722537 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:14.722793 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:14.722800 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:14.722856 (+ 56us) client_negotiation.cc:769] Sending connection context
0926 20:49:14.722955 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 20:49:14.723077 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":91,"threads_started":1}
I20260926 20:49:14.723721 6496 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:44089: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "27e19353428e42f09a3f45b96fb8a573"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 44089
}
attrs {
promote: true
}
}
dest_uuid: "4e93f52007174325a60947bfdcddc4d0"
cas_config_opid_index: -1
I20260926 20:49:14.724095 6496 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44089 } attrs { promote: true } }
I20260926 20:49:14.724597 7055 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 27e19353428e42f09a3f45b96fb8a573 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44089 } attrs { promote: true } } }
I20260926 20:49:14.724870 7002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:49:14.725759 7055 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:44089
I20260926 20:49:14.725809 7002 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573 [term 1 FOLLOWER]: Refusing update from remote peer 4e93f52007174325a60947bfdcddc4d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 20:49:14.726008 6960 master.cc:537] Master@127.0.98.61:44089 shutting down...
I20260926 20:49:14.725996 7055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4e93f52007174325a60947bfdcddc4d0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44089 } attrs { promote: true } } }
I20260926 20:49:14.726063 7055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0 [sys.catalog]: This master's current role is: LEADER
I20260926 20:49:14.726095 7054 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44089 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 20:49:14.726755 6482 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260926 20:49:14.727183 6482 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0 -> Peer 27e19353428e42f09a3f45b96fb8a573 (127.0.98.61:44089): Couldn't send request to peer 27e19353428e42f09a3f45b96fb8a573. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 20:49:14.728911 6960 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:49:14.729005 6960 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:49:14.729075 6960 tablet_replica.cc:330] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573: stopping tablet replica
I20260926 20:49:14.741557 6960 master.cc:552] Master@127.0.98.61:44089 shutdown complete.
I20260926 20:49:14.741608 6960 master_runner.cc:305] Clearing existing system tablet
I20260926 20:49:14.741756 6960 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:14.741808 6960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:14.742444 6960 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260926 20:49:14.742959 7060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:14.743070 6960 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:14.743129 6960 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:14.743150 6960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-1/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-1/wal
uuid: "27e19353428e42f09a3f45b96fb8a573"
format_stamp: "Formatted at 2026-09-26 20:49:14 on dist-test-slave-hwlj"
server_key: "5b196f2d96fb00870cf7f1d89d81144a"
server_key_iv: "7d88f0d4e486e79dc84256a5d46a3129"
server_key_version: "encryptionkey@0"
I20260926 20:49:14.743947 6960 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:49:14.745631 6960 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 20:49:14.745712 6960 log.cc:1196] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 20:49:14.745929 6960 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573: Deleting consensus metadata
I20260926 20:49:14.746061 6960 master_runner.cc:315] Copying system tablet from 127.0.98.62:40969
I20260926 20:49:14.746685 6960 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:40969
I20260926 20:49:14.753219 7066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:14.747007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40969 (local address 127.0.0.1:43488)
0926 20:49:14.747139 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:14.747151 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:14.747227 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:14.747360 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:14.747361 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:14.747519 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:14.747583 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:14.747585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:14.748607 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:14.748608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:14.749172 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:14.749174 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:14.749643 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:14.749644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:14.749690 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:14.750194 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:14.750198 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:14.750488 (+ 290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:14.752217 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:14.752218 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:14.752218 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:14.752392 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:14.752605 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:14.752606 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:14.752606 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:14.752659 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:14.752936 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:14.752936 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:14.752993 (+ 57us) client_negotiation.cc:769] Sending connection context
0926 20:49:14.753061 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 20:49:14.753142 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":78,"threads_started":1}
I20260926 20:49:14.753270 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:14.747103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43488 (local address 127.0.98.62:40969)
0926 20:49:14.747164 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:14.747166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:14.747212 (+ 46us) server_negotiation.cc:407] Connection header received
0926 20:49:14.747286 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:14.747287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:14.747299 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:14.747356 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:14.747659 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:14.748516 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:14.749252 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:14.749570 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:14.749640 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:14.750589 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:14.750590 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:14.750590 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:14.750598 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:14.752108 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:14.752470 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:14.752471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:14.752472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:14.752530 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:14.752721 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:14.752721 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:14.752722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:14.752850 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:14.752918 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:14.753103 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 20:49:14.753176 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 20:49:14.753881 6516 tablet_copy_service.cc:140] P 4e93f52007174325a60947bfdcddc4d0: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 27e19353428e42f09a3f45b96fb8a573 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:43488)
I20260926 20:49:14.753988 6516 tablet_copy_service.cc:161] P 4e93f52007174325a60947bfdcddc4d0: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 27e19353428e42f09a3f45b96fb8a573 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:43488: session id = 27e19353428e42f09a3f45b96fb8a573-00000000000000000000000000000000
I20260926 20:49:14.754820 6516 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 20:49:14.755470 6960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 20:49:14.756520 6960 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573: tablet copy: Starting download of 0 data blocks...
I20260926 20:49:14.756681 6960 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573: tablet copy: Starting download of 1 WAL segments...
I20260926 20:49:14.757496 6960 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 20:49:14.758275 6516 tablet_copy_service.cc:342] P 4e93f52007174325a60947bfdcddc4d0: Request end of tablet copy session 27e19353428e42f09a3f45b96fb8a573-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:43488
I20260926 20:49:14.758327 6516 tablet_copy_service.cc:433] P 4e93f52007174325a60947bfdcddc4d0: ending tablet copy session 27e19353428e42f09a3f45b96fb8a573-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 27e19353428e42f09a3f45b96fb8a573
I20260926 20:49:14.760694 6960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:14.761766 7067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:14.761891 6960 server_base.cc:1034] running on GCE node
W20260926 20:49:14.761907 7068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:14.761766 7070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:14.762107 6960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:14.762266 6960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:14.763412 6960 hybrid_clock.cc:648] HybridClock initialized: now 1790455754763400 us; error 27 us; skew 500 ppm
Sep 26 20:49:14 dist-test-slave-hwlj krb5kdc[6451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455754, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:49:14.764983 6960 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260926 20:49:14.765273 6960 webserver.cc:460] Webserver started at http://127.0.98.61:44619/ using document root <none> and password file <none>
I20260926 20:49:14.765390 6960 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:14.765445 6960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:14.766094 6960 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:49:14.766601 7077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:14.766760 6960 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:49:14.766819 6960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-1/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-1/wal
uuid: "27e19353428e42f09a3f45b96fb8a573"
format_stamp: "Formatted at 2026-09-26 20:49:14 on dist-test-slave-hwlj"
server_key: "5b196f2d96fb00870cf7f1d89d81144a"
server_key_iv: "7d88f0d4e486e79dc84256a5d46a3129"
server_key_version: "encryptionkey@0"
I20260926 20:49:14.766876 6960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:14.804352 6960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:14.806519 6960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:14.809727 6960 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:44089
I20260926 20:49:14.809759 7129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:44089 every 8 connection(s)
I20260926 20:49:14.810070 6960 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-1/data/info.pb
I20260926 20:49:14.810781 7130 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:49:14.811192 7130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573: Bootstrap starting.
I20260926 20:49:14.813534 7130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:49:14.813759 7130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573: Bootstrap complete.
I20260926 20:49:14.813969 7130 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44089 } attrs { promote: true } }
I20260926 20:49:14.814061 7130 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 27e19353428e42f09a3f45b96fb8a573, State: Initialized, Role: LEARNER
I20260926 20:49:14.814107 7130 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44089 } attrs { promote: true } }
I20260926 20:49:14.814301 7130 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:49:14.814291 7133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44089 } attrs { promote: true } } }
I20260926 20:49:14.814373 7133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573 [sys.catalog]: This master's current role is: LEARNER
I20260926 20:49:14.815796 7144 catalog_manager.cc:1259] Loaded cluster ID: d234dc66a6104be1b78c57501e0b0d6d
I20260926 20:49:14.815876 7144 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573: loading cluster ID for follower catalog manager: success
I20260926 20:49:14.817034 7144 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573: acquiring CA information for follower catalog manager: success
I20260926 20:49:14.817333 7144 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 20:49:14.848304 7145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:14.841163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37122 (local address 127.0.98.61:44089)
0926 20:49:14.841282 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:14.841284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:14.841330 (+ 46us) server_negotiation.cc:407] Connection header received
0926 20:49:14.841359 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:14.841359 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:14.841372 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:14.841445 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:14.842003 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:14.843040 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:14.843794 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:14.844089 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:14.844157 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:14.845508 (+ 1351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:14.845510 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:14.845510 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:14.845518 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:14.847121 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:14.847460 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:14.847461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:14.847462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:14.847556 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:14.847769 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:14.847770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:14.847771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:14.847901 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:14.848003 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:14.848097 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 20:49:14.848177 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":66,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260926 20:49:14.875118 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41871
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40969,127.0.98.61:44089,127.0.98.60:41871
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:49:14.956106 7148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:14.956317 7148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:14.956339 7148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:14.957527 7148 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:49:14.957571 7148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:14.957587 7148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:14.957599 7148 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:49:14.957647 7148 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:49:14.959148 7148 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40969,127.0.98.61:44089,127.0.98.60:41871
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:14.959399 7148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:14.959589 7148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:14.964550 7156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:14.964550 7154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:14.964644 7153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:14.964875 7148 server_base.cc:1034] running on GCE node
I20260926 20:49:14.965111 7148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:14.965444 7148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:14.966629 7148 hybrid_clock.cc:648] HybridClock initialized: now 1790455754966623 us; error 29 us; skew 500 ppm
Sep 26 20:49:14 dist-test-slave-hwlj krb5kdc[6451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455754, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:49:14.969110 7148 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260926 20:49:14.969605 7148 webserver.cc:460] Webserver started at http://127.0.98.60:45521/ using document root <none> and password file <none>
I20260926 20:49:14.969846 7148 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:14.969913 7148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:14.970062 7148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:49:14.971107 7148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-2/data/instance:
uuid: "a19b05eba1c24e7db3c2ad66aa45fd33"
format_stamp: "Formatted at 2026-09-26 20:49:14 on dist-test-slave-hwlj"
server_key: "4f5f40803525e9ea26f83415d6bdeba5"
server_key_iv: "2858b09fc77d6208ce812825306f73d3"
server_key_version: "encryptionkey@0"
I20260926 20:49:14.971485 7148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-2/wal/instance:
uuid: "a19b05eba1c24e7db3c2ad66aa45fd33"
format_stamp: "Formatted at 2026-09-26 20:49:14 on dist-test-slave-hwlj"
server_key: "4f5f40803525e9ea26f83415d6bdeba5"
server_key_iv: "2858b09fc77d6208ce812825306f73d3"
server_key_version: "encryptionkey@0"
I20260926 20:49:14.972905 7148 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:49:14.973636 7163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:14.973815 7148 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:14.973909 7148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-2/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-2/wal
uuid: "a19b05eba1c24e7db3c2ad66aa45fd33"
format_stamp: "Formatted at 2026-09-26 20:49:14 on dist-test-slave-hwlj"
server_key: "4f5f40803525e9ea26f83415d6bdeba5"
server_key_iv: "2858b09fc77d6208ce812825306f73d3"
server_key_version: "encryptionkey@0"
I20260926 20:49:14.973995 7148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:14.987385 7148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:14.989403 7148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:14.989565 7148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:14.994249 7148 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:41871
I20260926 20:49:14.994302 7215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:41871 every 8 connection(s)
I20260926 20:49:14.994714 7148 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-2/data/info.pb
I20260926 20:49:14.995728 7216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:49:14.997404 7216 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:49:14.999946 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 7148
Sep 26 20:49:15 dist-test-slave-hwlj krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455754, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 26 20:49:15 dist-test-slave-hwlj krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260926 20:49:15.007908 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:14.998217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43504 (local address 127.0.98.62:40969)
0926 20:49:14.998288 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:14.998290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:14.998409 (+ 119us) server_negotiation.cc:407] Connection header received
0926 20:49:14.998845 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:14.998846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:14.998856 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:14.998907 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:14.999453 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.000020 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.001583 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.001871 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.001946 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.004503 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:15.004504 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:15.004505 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:15.004545 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:15.006157 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:15.006684 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:15.006686 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:15.006686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:15.006750 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:15.007122 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:15.007123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:15.007123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:15.007276 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:15.007345 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:15.007678 (+ 333us) server_negotiation.cc:299] Negotiation successful
0926 20:49:15.007795 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 20:49:15.008123 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:14.998074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40969 (local address 127.0.0.1:43504)
0926 20:49:14.998294 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:14.998330 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:14.998764 (+ 434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:14.998958 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:14.998961 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:14.999205 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:14.999387 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:14.999391 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.000128 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.000129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:15.001496 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:15.001499 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.001970 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.001971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:15.002091 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.002634 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:15.002644 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:15.004422 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:15.006267 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:15.006273 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:15.006274 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:15.006571 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:15.006817 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:15.006817 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:15.006818 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:15.007046 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:15.007358 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:15.007365 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:15.007528 (+ 163us) client_negotiation.cc:769] Sending connection context
0926 20:49:15.007654 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 20:49:15.007831 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":96,"threads_started":1}
I20260926 20:49:15.008687 7216 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44089 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:49:15.010113 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:15.000462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35678 (local address 127.0.98.60:41871)
0926 20:49:15.000630 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:15.000634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:15.000648 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:49:15.000693 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:15.000695 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:15.000711 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:15.000772 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:15.001455 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.002379 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.003493 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.003990 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.004064 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.006755 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:15.006760 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:15.006760 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:15.006765 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:15.008918 (+ 2153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:15.009357 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:15.009359 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:15.009360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:15.009439 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:15.009604 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:15.009604 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:15.009605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:15.009761 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:15.009910 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:15.009949 (+ 39us) server_negotiation.cc:299] Negotiation successful
0926 20:49:15.010019 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":110,"threads_started":1}
Sep 26 20:49:15 dist-test-slave-hwlj krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455754, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 20:49:15.015312 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:15.008931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44089 (local address 127.0.0.1:37128)
0926 20:49:15.008978 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:15.008988 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:15.009029 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:15.009172 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:15.009173 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:15.009311 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:15.009348 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:15.009350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.010241 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.010242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:15.010953 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:15.010955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.011380 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.011381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:15.011439 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.011800 (+ 361us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:15.011804 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:15.012859 (+ 1055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:15.014345 (+ 1486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:15.014346 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:15.014346 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:15.014496 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:15.014710 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:15.014711 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:15.014712 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:15.014769 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:15.014978 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:15.014978 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:15.015032 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 20:49:15.015119 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 20:49:15.015223 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 20:49:15.015321 7145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:15.009002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37128 (local address 127.0.98.61:44089)
0926 20:49:15.009057 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:15.009058 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:15.009066 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:49:15.009092 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:15.009093 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:15.009112 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:15.009152 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:15.009425 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.010164 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.011038 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.011293 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.011370 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.012947 (+ 1577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:15.012949 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:15.012950 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:15.012957 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:15.014204 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:15.014559 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:15.014560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:15.014560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:15.014609 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:15.014823 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:15.014824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:15.014824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:15.014922 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:15.015002 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:15.015150 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 20:49:15.015233 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 20:49:15.016027 7216 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41871 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:49:15 dist-test-slave-hwlj krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455754, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260926 20:49:15.021754 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:15.016293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41871 (local address 127.0.0.1:35680)
0926 20:49:15.016362 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:15.016372 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:15.016403 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:15.016515 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:15.016516 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:15.016646 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:15.016671 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:15.016673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.017304 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.017306 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:15.017794 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:15.017796 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.018173 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.018174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:15.018220 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.018531 (+ 311us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:15.018535 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:15.019574 (+ 1039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:15.021003 (+ 1429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:15.021004 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:15.021004 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:15.021154 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:15.021319 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:15.021320 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:15.021320 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:15.021359 (+ 39us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:15.021547 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:15.021547 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:15.021577 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 20:49:15.021625 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 20:49:15.021699 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 20:49:15.021879 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:15.016332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35680 (local address 127.0.98.60:41871)
0926 20:49:15.016386 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:15.016387 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:15.016410 (+ 23us) server_negotiation.cc:407] Connection header received
0926 20:49:15.016461 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:15.016461 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:15.016470 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:15.016514 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:15.016755 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.017223 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.017888 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.018104 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.018176 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.019657 (+ 1481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:15.019658 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:15.019659 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:15.019665 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:15.020932 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:15.021229 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:15.021230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:15.021231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:15.021278 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:15.021406 (+ 128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:15.021406 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:15.021407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:15.021492 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:15.021540 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:15.021642 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 20:49:15.021744 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":34}
I20260926 20:49:15.023042 7216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33: Bootstrap starting.
I20260926 20:49:15.023859 7216 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33: Neither blocks nor log segments found. Creating new log.
I20260926 20:49:15.024145 7216 log.cc:824] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33: Log is configured to *not* fsync() on all Append() calls
I20260926 20:49:15.024808 7216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33: No bootstrap required, opened a new log
I20260926 20:49:15.026144 7216 raft_consensus.cc:348] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44089 } } peers { permanent_uuid: "a19b05eba1c24e7db3c2ad66aa45fd33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41871 } }
I20260926 20:49:15.026266 7216 raft_consensus.cc:374] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:49:15.026288 7216 raft_consensus.cc:729] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a19b05eba1c24e7db3c2ad66aa45fd33, State: Initialized, Role: FOLLOWER
I20260926 20:49:15.026425 7216 consensus_queue.cc:260] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44089 } } peers { permanent_uuid: "a19b05eba1c24e7db3c2ad66aa45fd33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41871 } }
I20260926 20:49:15.027148 7216 sys_catalog.cc:564] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:49:15.027138 7221 sys_catalog.cc:455] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44089 } } peers { permanent_uuid: "a19b05eba1c24e7db3c2ad66aa45fd33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41871 } } }
I20260926 20:49:15.027251 7221 sys_catalog.cc:458] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 20:49:15.028843 7232 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:49:15.029034 7232 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:49:15.031162 7148 master_runner.cc:428] Detected that this master a19b05eba1c24e7db3c2ad66aa45fd33 is joining an existing cluster
I20260926 20:49:15.031213 7148 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:41871
I20260926 20:49:15.039350 7240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:15.032543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41871 (local address 127.0.0.1:35692)
0926 20:49:15.032739 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:15.032749 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:15.032807 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:15.032956 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:15.032957 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:15.033348 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:15.033406 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:15.033407 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.034315 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.034315 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:15.035235 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:15.035238 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.035732 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.035733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:15.035823 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.036338 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:15.036343 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:15.036692 (+ 349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:15.038411 (+ 1719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:15.038412 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:15.038413 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:15.038605 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:15.038812 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:15.038813 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:15.038813 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:15.038866 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:15.039106 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:15.039106 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:15.039140 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 20:49:15.039202 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 20:49:15.039278 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":2,"thread_start_us":71,"threads_started":1}
I20260926 20:49:15.039443 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:15.032568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35692 (local address 127.0.98.60:41871)
0926 20:49:15.032603 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:15.032605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:15.032788 (+ 183us) server_negotiation.cc:407] Connection header received
0926 20:49:15.032861 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:15.032861 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:15.032879 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:15.032904 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:15.033525 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.034217 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.035335 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.035623 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.035661 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.036773 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:15.036774 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:15.036775 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:15.036782 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:15.038330 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:15.038686 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:15.038687 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:15.038688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:15.038746 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:15.038924 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:15.038925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:15.038925 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:15.039045 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:15.039110 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:15.039237 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 20:49:15.039338 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":20}
I20260926 20:49:15.061462 7221 raft_consensus.cc:482] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:49:15.061555 7221 raft_consensus.cc:504] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44089 } } peers { permanent_uuid: "a19b05eba1c24e7db3c2ad66aa45fd33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41871 } }
I20260926 20:49:15.061789 7221 leader_election.cc:288] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e93f52007174325a60947bfdcddc4d0 (127.0.98.62:40969), 27e19353428e42f09a3f45b96fb8a573 (127.0.98.61:44089)
I20260926 20:49:15.062058 6506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a19b05eba1c24e7db3c2ad66aa45fd33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e93f52007174325a60947bfdcddc4d0" is_pre_election: true
I20260926 20:49:15.062287 7104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a19b05eba1c24e7db3c2ad66aa45fd33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27e19353428e42f09a3f45b96fb8a573" is_pre_election: true
I20260926 20:49:15.062404 7104 raft_consensus.cc:1799] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573 [term 1 LEARNER]: Handling vote request from an unknown peer a19b05eba1c24e7db3c2ad66aa45fd33
I20260926 20:49:15.062403 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 6558
I20260926 20:49:15.062455 7104 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573 [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate a19b05eba1c24e7db3c2ad66aa45fd33 for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260926 20:49:15.063127 7164 leader_election.cc:302] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a19b05eba1c24e7db3c2ad66aa45fd33; no voters: 27e19353428e42f09a3f45b96fb8a573, 4e93f52007174325a60947bfdcddc4d0
I20260926 20:49:15.063306 7221 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:49:15.064374 7221 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 20:49:15.064634 7145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:15.032392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37142 (local address 127.0.98.61:44089)
0926 20:49:15.032440 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:15.032442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:15.032751 (+ 309us) server_negotiation.cc:407] Connection header received
0926 20:49:15.032845 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:15.032846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:15.032856 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:15.032886 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:15.033468 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.034219 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.035405 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.035707 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.035775 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.061095 (+ 25320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:15.061098 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:15.061098 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:15.061107 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:15.062911 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:15.063689 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:15.063691 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:15.063691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:15.063751 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:15.063983 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:15.063984 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:15.063984 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:15.064140 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:15.064188 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:15.064404 (+ 216us) server_negotiation.cc:299] Negotiation successful
0926 20:49:15.064499 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 20:49:15.065044 7239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:15.032279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44089 (local address 127.0.0.1:37142)
0926 20:49:15.032687 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:15.032698 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:15.032775 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:15.032949 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:15.032950 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:15.033296 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:15.033373 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:15.033375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.034315 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.034316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:15.035304 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:15.035306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.035861 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.035862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:15.035924 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.060394 (+ 24470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:15.060399 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:15.060970 (+ 571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:15.063334 (+ 2364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:15.063335 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:15.063335 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:15.063596 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:15.063853 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:15.063853 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:15.063854 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:15.063915 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:15.064239 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:15.064240 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:15.064288 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 20:49:15.064748 (+ 460us) client_negotiation.cc:240] Negotiation successful
0926 20:49:15.064876 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"mutex_wait_us":28,"thread_start_us":415,"threads_started":1}
I20260926 20:49:15.068059 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46533
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40969
--webserver_port=33941
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:40969,127.0.98.61:44089
--tserver_master_addrs=127.0.98.62:40969,127.0.98.61:44089,127.0.98.60:41871 with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:49:15.068239 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:15.032329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43514 (local address 127.0.98.62:40969)
0926 20:49:15.032379 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:15.032381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:15.032492 (+ 111us) server_negotiation.cc:407] Connection header received
0926 20:49:15.032678 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:15.032679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:15.032690 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:15.032724 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:15.033482 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.034226 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.035347 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.035652 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.035688 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.061136 (+ 25448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:15.061139 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:15.061139 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:15.061149 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:15.066822 (+ 5673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:15.067307 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:15.067308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:15.067309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:15.067367 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:15.067608 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:15.067609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:15.067610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:15.067758 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:15.067827 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:15.068062 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 20:49:15.068147 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 20:49:15.068267 7238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:15.032277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40969 (local address 127.0.0.1:43514)
0926 20:49:15.032432 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:15.032443 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:15.032520 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:15.032934 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:15.032935 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:15.033318 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:15.033383 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:15.033385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.034309 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.034309 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:15.035260 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:15.035262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.035784 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.035785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:15.035838 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.060459 (+ 24621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:15.060463 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:15.061003 (+ 540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:15.066936 (+ 5933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:15.066937 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:15.066937 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:15.067220 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:15.067442 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:15.067443 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:15.067443 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:15.067519 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:15.067884 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:15.067884 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:15.067932 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 20:49:15.068003 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 20:49:15.068120 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":53,"thread_start_us":77,"threads_started":1}
Sep 26 20:49:15 dist-test-slave-hwlj krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455753, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260926 20:49:15.078619 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:15.070386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41871 (local address 127.0.0.1:35698)
0926 20:49:15.070434 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:15.070443 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:15.070475 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:15.070602 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:15.070603 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:15.070765 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:15.070809 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:15.070811 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.071733 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.071734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:15.072710 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:15.072713 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.073272 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.073273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:15.073348 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.073774 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:15.073780 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:15.075335 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:15.077407 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:15.077409 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:15.077409 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:15.077703 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:15.077928 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:15.077929 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:15.077929 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:15.077996 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:15.078291 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:15.078292 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:15.078338 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 20:49:15.078415 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 20:49:15.078514 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 20:49:15.079034 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:15.070445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35698 (local address 127.0.98.60:41871)
0926 20:49:15.070483 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:15.070485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:15.070496 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:49:15.070531 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:15.070532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:15.070543 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:15.070601 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:15.070883 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.071632 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.072813 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.073151 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.073218 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.075423 (+ 2205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:15.075425 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:15.075425 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:15.075433 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:15.077305 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:15.077781 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:15.077782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:15.077783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:15.077845 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:15.078066 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:15.078067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:15.078067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:15.078214 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:15.078291 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:15.078461 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 20:49:15.078538 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 20:49:15.079710 6496 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:41871: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a19b05eba1c24e7db3c2ad66aa45fd33"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 41871
}
attrs {
promote: true
}
}
dest_uuid: "4e93f52007174325a60947bfdcddc4d0"
cas_config_opid_index: 5
I20260926 20:49:15.079977 6496 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44089 } attrs { promote: true } } peers { permanent_uuid: "a19b05eba1c24e7db3c2ad66aa45fd33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41871 } attrs { promote: true } }
I20260926 20:49:15.080325 7056 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a19b05eba1c24e7db3c2ad66aa45fd33 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44089 } attrs { promote: true } } peers { permanent_uuid: "a19b05eba1c24e7db3c2ad66aa45fd33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41871 } attrs { promote: true } } }
I20260926 20:49:15.080703 7190 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [term 1 FOLLOWER]: Refusing update from remote peer 4e93f52007174325a60947bfdcddc4d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 20:49:15.081281 7056 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:41871
I20260926 20:49:15.081352 7056 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a19b05eba1c24e7db3c2ad66aa45fd33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41871 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:49:15.081563 7056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4e93f52007174325a60947bfdcddc4d0" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44089 } attrs { promote: true } } peers { permanent_uuid: "a19b05eba1c24e7db3c2ad66aa45fd33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41871 } attrs { promote: true } } }
I20260926 20:49:15.081629 7056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0 [sys.catalog]: This master's current role is: LEADER
I20260926 20:49:15.082017 7148 master.cc:537] Master@127.0.98.60:41871 shutting down...
I20260926 20:49:15.082644 7190 raft_consensus.cc:858] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44089 } } peers { permanent_uuid: "a19b05eba1c24e7db3c2ad66aa45fd33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41871 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44089 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44089 } attrs { promote: true } } peers { permanent_uuid: "a19b05eba1c24e7db3c2ad66aa45fd33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41871 } attrs { promote: true } }
W20260926 20:49:15.082695 7190 raft_consensus.cc:1565] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44089 } } peers { permanent_uuid: "a19b05eba1c24e7db3c2ad66aa45fd33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41871 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44089 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44089 } attrs { promote: true } } peers { permanent_uuid: "a19b05eba1c24e7db3c2ad66aa45fd33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41871 } attrs { promote: true } }
I20260926 20:49:15.083150 7190 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 27e19353428e42f09a3f45b96fb8a573 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44089 } attrs { promote: true } } }
I20260926 20:49:15.084120 7221 sys_catalog.cc:455] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e93f52007174325a60947bfdcddc4d0. Latest consensus state: current_term: 1 leader_uuid: "4e93f52007174325a60947bfdcddc4d0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44089 } attrs { promote: true } } }
I20260926 20:49:15.084187 7221 sys_catalog.cc:458] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260926 20:49:15.084214 7221 sys_catalog.cc:455] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4e93f52007174325a60947bfdcddc4d0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44089 } attrs { promote: true } } }
I20260926 20:49:15.084247 7221 sys_catalog.cc:458] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260926 20:49:15.084270 7221 sys_catalog.cc:455] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4e93f52007174325a60947bfdcddc4d0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44089 } attrs { promote: true } } }
I20260926 20:49:15.084299 7221 sys_catalog.cc:458] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260926 20:49:15.085525 7242 mvcc.cc:205] Tried to move back new op lower bound from 7333706768625119232 to 7333706768529031168. Current Snapshot: MvccSnapshot[applied={T|T < 7333706768538836992}]
I20260926 20:49:15.087283 7054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4e93f52007174325a60947bfdcddc4d0" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44089 } attrs { promote: true } } peers { permanent_uuid: "a19b05eba1c24e7db3c2ad66aa45fd33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41871 } attrs { promote: true } } }
I20260926 20:49:15.087369 7054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0 [sys.catalog]: This master's current role is: LEADER
I20260926 20:49:15.087533 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:15.080387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44089 (local address 127.0.0.1:37148)
0926 20:49:15.080423 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:15.080434 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:15.080479 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:15.084102 (+ 3623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:15.084103 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:15.084105 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:49:15.084142 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:15.084144 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.085158 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.085159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:15.086388 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:15.086391 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.087280 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.087282 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:15.087351 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.087377 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 20:49:15.087427 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 20:49:15.087466 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260926 20:49:15.088044 7145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:15.081782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37148 (local address 127.0.98.61:44089)
0926 20:49:15.083318 (+ 1536us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:15.083320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:15.083329 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:49:15.083355 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:15.083356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:15.083381 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:15.083445 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:49:15.084228 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.084974 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.086482 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.087092 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.087167 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.087246 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:15.087893 (+ 647us) server_negotiation.cc:299] Negotiation successful
0926 20:49:15.087944 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1513}
W20260926 20:49:15.088135 6483 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0 -> Peer a19b05eba1c24e7db3c2ad66aa45fd33 (127.0.98.60:41871): Couldn't send request to peer a19b05eba1c24e7db3c2ad66aa45fd33. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 20:49:15.088320 7148 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260926 20:49:15.088395 7148 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260926 20:49:15.088425 7148 tablet_replica.cc:330] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33: stopping tablet replica
I20260926 20:49:15.088434 7104 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573 [term 1 LEARNER]: Refusing update from remote peer 4e93f52007174325a60947bfdcddc4d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 20:49:15.088668 7056 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44089 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 20:49:15.089221 7104 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a19b05eba1c24e7db3c2ad66aa45fd33 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44089 } attrs { promote: true } } peers { permanent_uuid: "a19b05eba1c24e7db3c2ad66aa45fd33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41871 } attrs { promote: true } } }
I20260926 20:49:15.090291 7133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e93f52007174325a60947bfdcddc4d0. Latest consensus state: current_term: 1 leader_uuid: "4e93f52007174325a60947bfdcddc4d0" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44089 } attrs { promote: true } } peers { permanent_uuid: "a19b05eba1c24e7db3c2ad66aa45fd33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41871 } attrs { promote: true } } }
I20260926 20:49:15.090365 7133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573 [sys.catalog]: This master's current role is: LEARNER
I20260926 20:49:15.090394 7133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4e93f52007174325a60947bfdcddc4d0" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44089 } attrs { promote: true } } peers { permanent_uuid: "a19b05eba1c24e7db3c2ad66aa45fd33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41871 } attrs { promote: true } } }
I20260926 20:49:15.090430 7133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573 [sys.catalog]: This master's current role is: LEARNER
I20260926 20:49:15.090742 7055 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0: attempting to promote NON_VOTER 27e19353428e42f09a3f45b96fb8a573 to VOTER
I20260926 20:49:15.090951 7055 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44089 } attrs { promote: false } } peers { permanent_uuid: "a19b05eba1c24e7db3c2ad66aa45fd33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41871 } attrs { promote: true } }
W20260926 20:49:15.091578 6483 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0 -> Peer a19b05eba1c24e7db3c2ad66aa45fd33 (127.0.98.60:41871): Couldn't send request to peer a19b05eba1c24e7db3c2ad66aa45fd33. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 20:49:15.091579 7104 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573 [term 1 LEARNER]: Refusing update from remote peer 4e93f52007174325a60947bfdcddc4d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 20:49:15.091806 7056 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260926 20:49:15.092453 7054 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 27e19353428e42f09a3f45b96fb8a573 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44089 } attrs { promote: false } } peers { permanent_uuid: "a19b05eba1c24e7db3c2ad66aa45fd33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41871 } attrs { promote: true } } }
I20260926 20:49:15.092590 7104 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 27e19353428e42f09a3f45b96fb8a573 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44089 } attrs { promote: false } } peers { permanent_uuid: "a19b05eba1c24e7db3c2ad66aa45fd33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41871 } attrs { promote: true } } }
I20260926 20:49:15.093387 7054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4e93f52007174325a60947bfdcddc4d0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44089 } attrs { promote: false } } peers { permanent_uuid: "a19b05eba1c24e7db3c2ad66aa45fd33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41871 } attrs { promote: true } } }
I20260926 20:49:15.093452 7054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0 [sys.catalog]: This master's current role is: LEADER
I20260926 20:49:15.093497 7054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4e93f52007174325a60947bfdcddc4d0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44089 } attrs { promote: false } } peers { permanent_uuid: "a19b05eba1c24e7db3c2ad66aa45fd33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41871 } attrs { promote: true } } }
I20260926 20:49:15.093477 7133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4e93f52007174325a60947bfdcddc4d0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44089 } attrs { promote: false } } peers { permanent_uuid: "a19b05eba1c24e7db3c2ad66aa45fd33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41871 } attrs { promote: true } } }
I20260926 20:49:15.093537 7054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0 [sys.catalog]: This master's current role is: LEADER
I20260926 20:49:15.093549 7133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:49:15.108215 7148 master.cc:552] Master@127.0.98.60:41871 shutdown complete.
I20260926 20:49:15.108279 7148 master_runner.cc:305] Clearing existing system tablet
I20260926 20:49:15.108407 7148 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:15.108454 7148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:15.109216 7148 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:49:15.109792 7252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:15.109943 7148 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:15.110013 7148 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:15.110037 7148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-2/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-2/wal
uuid: "a19b05eba1c24e7db3c2ad66aa45fd33"
format_stamp: "Formatted at 2026-09-26 20:49:14 on dist-test-slave-hwlj"
server_key: "4f5f40803525e9ea26f83415d6bdeba5"
server_key_iv: "2858b09fc77d6208ce812825306f73d3"
server_key_version: "encryptionkey@0"
I20260926 20:49:15.110737 7148 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:49:15.112282 7148 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 20:49:15.112353 7148 log.cc:1196] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 20:49:15.112571 7148 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33: Deleting consensus metadata
I20260926 20:49:15.112825 7148 master_runner.cc:315] Copying system tablet from 127.0.98.62:40969
I20260926 20:49:15.113431 7148 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:40969
I20260926 20:49:15.121027 7258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:15.113746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40969 (local address 127.0.0.1:43520)
0926 20:49:15.113873 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:15.113885 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:15.113950 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:15.114120 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:15.114121 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:15.114324 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:15.114382 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:15.114384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.115408 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.115409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:15.116301 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:15.116303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.116816 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.116817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:15.116883 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.117458 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:15.117463 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:15.117884 (+ 421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:15.119865 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:15.119866 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:15.119867 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:15.120091 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:15.120315 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:15.120316 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:15.120316 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:15.120388 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:15.120677 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:15.120678 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:15.120730 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 20:49:15.120812 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 20:49:15.120923 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":68,"threads_started":1}
I20260926 20:49:15.121037 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:15.113824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43520 (local address 127.0.98.62:40969)
0926 20:49:15.113872 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:15.113874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:15.113935 (+ 61us) server_negotiation.cc:407] Connection header received
0926 20:49:15.114017 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:15.114019 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:15.114030 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:15.114104 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:15.114466 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.115302 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.116387 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.116715 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.116800 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.117985 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:15.117986 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:15.117987 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:15.117994 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:15.119722 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:15.120173 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:15.120174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:15.120174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:15.120238 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:15.120459 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:15.120459 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:15.120460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:15.120593 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:15.120658 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:15.120847 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 20:49:15.120928 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 20:49:15.121704 6516 tablet_copy_service.cc:140] P 4e93f52007174325a60947bfdcddc4d0: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a19b05eba1c24e7db3c2ad66aa45fd33 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:43520)
I20260926 20:49:15.121767 6516 tablet_copy_service.cc:161] P 4e93f52007174325a60947bfdcddc4d0: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a19b05eba1c24e7db3c2ad66aa45fd33 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:43520: session id = a19b05eba1c24e7db3c2ad66aa45fd33-00000000000000000000000000000000
I20260926 20:49:15.122339 6516 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 20:49:15.122790 7148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 20:49:15.124106 7148 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33: tablet copy: Starting download of 0 data blocks...
I20260926 20:49:15.124248 7148 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33: tablet copy: Starting download of 1 WAL segments...
I20260926 20:49:15.125036 7148 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 20:49:15.125903 6516 tablet_copy_service.cc:342] P 4e93f52007174325a60947bfdcddc4d0: Request end of tablet copy session a19b05eba1c24e7db3c2ad66aa45fd33-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:43520
I20260926 20:49:15.125962 6516 tablet_copy_service.cc:433] P 4e93f52007174325a60947bfdcddc4d0: ending tablet copy session a19b05eba1c24e7db3c2ad66aa45fd33-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a19b05eba1c24e7db3c2ad66aa45fd33
I20260926 20:49:15.128695 7148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:15.129894 7260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:15.129902 7259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:15.130371 7262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:15.130398 7148 server_base.cc:1034] running on GCE node
I20260926 20:49:15.130600 7148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:15.130735 7148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:15.131880 7148 hybrid_clock.cc:648] HybridClock initialized: now 1790455755131858 us; error 40 us; skew 500 ppm
Sep 26 20:49:15 dist-test-slave-hwlj krb5kdc[6451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455755, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:49:15.133365 7148 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260926 20:49:15.133622 7148 webserver.cc:460] Webserver started at http://127.0.98.60:43499/ using document root <none> and password file <none>
I20260926 20:49:15.133752 7148 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:15.133790 7148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:15.134506 7148 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:49:15.135094 7269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:15.135249 7148 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:49:15.135304 7148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-2/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-2/wal
uuid: "a19b05eba1c24e7db3c2ad66aa45fd33"
format_stamp: "Formatted at 2026-09-26 20:49:14 on dist-test-slave-hwlj"
server_key: "4f5f40803525e9ea26f83415d6bdeba5"
server_key_iv: "2858b09fc77d6208ce812825306f73d3"
server_key_version: "encryptionkey@0"
I20260926 20:49:15.135411 7148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:15.160252 7148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:15.162192 7148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:15.165259 7148 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:41871
I20260926 20:49:15.165298 7321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:41871 every 8 connection(s)
I20260926 20:49:15.165623 7148 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/master-2/data/info.pb
I20260926 20:49:15.166543 7322 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:49:15.167116 7322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33: Bootstrap starting.
I20260926 20:49:15.169505 7322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:49:15.169759 7322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33: Bootstrap complete.
I20260926 20:49:15.169960 7322 raft_consensus.cc:348] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44089 } attrs { promote: false } } peers { permanent_uuid: "a19b05eba1c24e7db3c2ad66aa45fd33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41871 } attrs { promote: true } }
I20260926 20:49:15.170050 7322 raft_consensus.cc:729] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a19b05eba1c24e7db3c2ad66aa45fd33, State: Initialized, Role: LEARNER
I20260926 20:49:15.170089 7322 consensus_queue.cc:260] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44089 } attrs { promote: false } } peers { permanent_uuid: "a19b05eba1c24e7db3c2ad66aa45fd33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41871 } attrs { promote: true } }
I20260926 20:49:15.170261 7322 sys_catalog.cc:564] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:49:15.170272 7325 sys_catalog.cc:455] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44089 } attrs { promote: false } } peers { permanent_uuid: "a19b05eba1c24e7db3c2ad66aa45fd33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41871 } attrs { promote: true } } }
I20260926 20:49:15.170482 7325 sys_catalog.cc:458] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [sys.catalog]: This master's current role is: LEARNER
I20260926 20:49:15.171718 7336 catalog_manager.cc:1259] Loaded cluster ID: d234dc66a6104be1b78c57501e0b0d6d
I20260926 20:49:15.171842 7336 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33: loading cluster ID for follower catalog manager: success
I20260926 20:49:15.173091 7336 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33: acquiring CA information for follower catalog manager: success
I20260926 20:49:15.173447 7336 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 20:49:15.174590 7241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:15.174805 7241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:15.174850 7241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:15.176080 7241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:15.176141 7241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:15.176187 7241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:49:15.177778 7241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46533
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33941
--tserver_master_addrs=127.0.98.62:40969,127.0.98.61:44089,127.0.98.60:41871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:15.178035 7241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:15.178261 7241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:15.184405 7342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:15.184424 7341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:15.184739 7241 server_base.cc:1034] running on GCE node
W20260926 20:49:15.184476 7344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:15.184979 7241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:15.185262 7241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:15.186410 7241 hybrid_clock.cc:648] HybridClock initialized: now 1790455755186376 us; error 50 us; skew 500 ppm
Sep 26 20:49:15 dist-test-slave-hwlj krb5kdc[6451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455755, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:49:15.189278 7241 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260926 20:49:15.189868 7241 webserver.cc:460] Webserver started at http://127.0.98.1:33941/ using document root <none> and password file <none>
I20260926 20:49:15.190135 7241 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:15.190196 7241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:15.191576 7241 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 20:49:15.192237 7351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:15.192411 7241 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 20:49:15.192508 7241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-0/wal
uuid: "e728cd363cea4949a2a5d07380f215d8"
format_stamp: "Formatted at 2026-09-26 20:49:14 on dist-test-slave-hwlj"
server_key: "b3d6bddd0112895917eb6c1122194e20"
server_key_iv: "26ddcba06655abf785ef003e648f7999"
server_key_version: "encryptionkey@0"
I20260926 20:49:15.192874 7241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:15.203102 7241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:15.205559 7241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:15.205682 7241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:15.205875 7241 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:49:15.206249 7241 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:49:15.206286 7241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:15.206310 7241 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:49:15.206352 7241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:15.212741 7241 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46533
I20260926 20:49:15.212798 7464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46533 every 8 connection(s)
I20260926 20:49:15.213160 7241 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:49:15.214402 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 7241
I20260926 20:49:15.214577 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 6692
I20260926 20:49:15.221279 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44443
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40969
--webserver_port=43457
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:40969,127.0.98.61:44089
--tserver_master_addrs=127.0.98.62:40969,127.0.98.61:44089,127.0.98.60:41871 with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:49:15 dist-test-slave-hwlj krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455755, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 26 20:49:15 dist-test-slave-hwlj krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455755, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 26 20:49:15 dist-test-slave-hwlj krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455755, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260926 20:49:15.226285 7145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:15.215031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33069 (local address 127.0.98.61:44089)
0926 20:49:15.217315 (+ 2284us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:15.217317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:15.217326 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:49:15.217351 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:15.217352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:15.217361 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:15.217412 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:15.217945 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.218682 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.219603 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.219897 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.219956 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.222497 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:15.222498 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:15.222499 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:15.222507 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:15.224336 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:15.224862 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:15.224863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:15.224864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:15.224920 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:15.225235 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:15.225236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:15.225237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:15.225427 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:15.225496 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:15.226119 (+ 623us) server_negotiation.cc:299] Negotiation successful
0926 20:49:15.226201 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2255}
I20260926 20:49:15.226352 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:15.213907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44089 (local address 127.0.98.1:33069)
0926 20:49:15.214101 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:15.214149 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:15.215412 (+ 1263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:15.217454 (+ 2042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:15.217455 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:15.217617 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:15.217674 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:15.217675 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.218771 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.218772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:15.219514 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:15.219515 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.220008 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.220009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:15.220070 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.220713 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:15.220718 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:15.222386 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:15.224428 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:15.224435 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:15.224438 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:15.224770 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:15.224998 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:15.224998 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:15.224999 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:15.225135 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:15.225533 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:15.225540 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:15.225821 (+ 281us) client_negotiation.cc:769] Sending connection context
0926 20:49:15.225933 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 20:49:15.226118 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":89,"threads_started":1}
I20260926 20:49:15.226797 7470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:15.213917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41871 (local address 127.0.98.1:48253)
0926 20:49:15.215598 (+ 1681us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:15.215606 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:15.215668 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:15.215976 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:15.215980 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:15.216468 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:15.216775 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:15.216780 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.218030 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.218031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:15.219224 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:15.219226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.219739 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.219740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:15.219894 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.220504 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:15.220514 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:15.223459 (+ 2945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:15.225461 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:15.225462 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:15.225462 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:15.225768 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:15.226027 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:15.226028 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:15.226028 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:15.226108 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:15.226411 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:15.226412 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:15.226459 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 20:49:15.226528 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 20:49:15.226644 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1667,"mutex_wait_us":14,"thread_start_us":323,"threads_started":1}
I20260926 20:49:15.227144 7467 heartbeater.cc:347] Connected to a master server at 127.0.98.61:44089
I20260926 20:49:15.227270 7471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:15.214064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48253 (local address 127.0.98.60:41871)
0926 20:49:15.214203 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:15.214205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:15.215662 (+ 1457us) server_negotiation.cc:407] Connection header received
0926 20:49:15.215735 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:15.215736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:15.215771 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:15.215838 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:15.216889 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.217948 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.219310 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.219627 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.219681 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.223534 (+ 3853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:15.223536 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:15.223536 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:15.223544 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:15.225349 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:15.225887 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:15.225888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:15.225888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:15.225952 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:15.226177 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:15.226178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:15.226178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:15.226321 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:15.226384 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:15.227012 (+ 628us) server_negotiation.cc:299] Negotiation successful
0926 20:49:15.227128 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":60,"threads_started":1}
I20260926 20:49:15.227285 7467 heartbeater.cc:464] Registering TS with master...
I20260926 20:49:15.227589 7465 heartbeater.cc:347] Connected to a master server at 127.0.98.60:41871
I20260926 20:49:15.227643 7465 heartbeater.cc:464] Registering TS with master...
I20260926 20:49:15.227730 7465 heartbeater.cc:511] Master 127.0.98.60:41871 requested a full tablet report, sending...
I20260926 20:49:15.227732 7467 heartbeater.cc:511] Master 127.0.98.61:44089 requested a full tablet report, sending...
I20260926 20:49:15.228276 7094 ts_manager.cc:194] Registered new tserver with Master: e728cd363cea4949a2a5d07380f215d8 (127.0.98.1:46533)
I20260926 20:49:15.228298 7286 ts_manager.cc:194] Registered new tserver with Master: e728cd363cea4949a2a5d07380f215d8 (127.0.98.1:46533)
I20260926 20:49:15.228677 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:15.214216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51543 (local address 127.0.98.62:40969)
0926 20:49:15.214266 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:15.214268 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:15.215837 (+ 1569us) server_negotiation.cc:407] Connection header received
0926 20:49:15.215937 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:15.215938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:15.215950 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:15.215978 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:15.216881 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.217727 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.219323 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.219674 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.219735 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.223629 (+ 3894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:15.223631 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:15.223631 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:15.223640 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:15.225830 (+ 2190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:15.227310 (+ 1480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:15.227312 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:15.227312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:15.227380 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:15.227604 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:15.227605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:15.227605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:15.227731 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:15.227821 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:15.228500 (+ 679us) server_negotiation.cc:299] Negotiation successful
0926 20:49:15.228582 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 20:49:15.228955 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:15.213907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40969 (local address 127.0.98.1:51543)
0926 20:49:15.215779 (+ 1872us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:15.215789 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:15.215861 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:15.216042 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:15.216043 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:15.216474 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:15.216774 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:15.216780 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.217857 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.217858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:15.219228 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:15.219231 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.219813 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.219814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:15.219900 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.220559 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:15.220564 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:15.223237 (+ 2673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:15.225920 (+ 2683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:15.225921 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:15.225921 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:15.227228 (+ 1307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:15.227479 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:15.227480 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:15.227480 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:15.227537 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:15.227860 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:15.227861 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:15.227911 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 20:49:15.228068 (+ 157us) client_negotiation.cc:240] Negotiation successful
0926 20:49:15.228878 (+ 810us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1818,"mutex_wait_us":74,"thread_start_us":113,"threads_started":1}
I20260926 20:49:15.229383 7466 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40969
I20260926 20:49:15.229424 7466 heartbeater.cc:464] Registering TS with master...
I20260926 20:49:15.229497 7466 heartbeater.cc:511] Master 127.0.98.62:40969 requested a full tablet report, sending...
I20260926 20:49:15.229782 6496 ts_manager.cc:194] Re-registered known tserver with Master: e728cd363cea4949a2a5d07380f215d8 (127.0.98.1:46533)
I20260926 20:49:15.230343 6496 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51543
W20260926 20:49:15.318269 7473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:15.318544 7473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:15.318580 7473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:15.320602 7473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:15.320649 7473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:15.320694 7473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:49:15.323001 7473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44443
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43457
--tserver_master_addrs=127.0.98.62:40969,127.0.98.61:44089,127.0.98.60:41871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:15.323264 7473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:15.323496 7473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:15.328770 7479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:15.328768 7478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:15.328977 7473 server_base.cc:1034] running on GCE node
W20260926 20:49:15.329002 7481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:15.329229 7473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:15.329502 7473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:15.330663 7473 hybrid_clock.cc:648] HybridClock initialized: now 1790455755330651 us; error 32 us; skew 500 ppm
Sep 26 20:49:15 dist-test-slave-hwlj krb5kdc[6451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455755, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:49:15.333349 7473 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260926 20:49:15.333863 7473 webserver.cc:460] Webserver started at http://127.0.98.2:43457/ using document root <none> and password file <none>
I20260926 20:49:15.334089 7473 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:15.334167 7473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:15.335675 7473 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:49:15.336287 7488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:15.336591 7473 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 20:49:15.336684 7473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-1/wal
uuid: "01cba43c9838497f8d037dec2c16e44a"
format_stamp: "Formatted at 2026-09-26 20:49:14 on dist-test-slave-hwlj"
server_key: "778f4dd9cd7606168935f47996d6d73f"
server_key_iv: "8466fedcfecf4a68ca6c2aeb966e6d75"
server_key_version: "encryptionkey@0"
I20260926 20:49:15.337074 7473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:15.361778 7473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:15.364183 7473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:15.364343 7473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:15.364596 7473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:49:15.364904 7473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:49:15.364938 7473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:15.364960 7473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:49:15.365005 7473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:15.371076 7473 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44443
I20260926 20:49:15.371129 7601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44443 every 8 connection(s)
I20260926 20:49:15.371496 7473 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
I20260926 20:49:15.377235 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 7473
I20260926 20:49:15.377393 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 6826
Sep 26 20:49:15 dist-test-slave-hwlj krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455755, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 26 20:49:15 dist-test-slave-hwlj krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455755, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 26 20:49:15 dist-test-slave-hwlj krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455755, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 20:49:15.383975 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:15.372938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34691 (local address 127.0.98.62:40969)
0926 20:49:15.373007 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:15.373009 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:15.373229 (+ 220us) server_negotiation.cc:407] Connection header received
0926 20:49:15.373577 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:15.373578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:15.373591 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:15.373637 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:15.374547 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.375046 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.376138 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.376347 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.376431 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.379095 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:15.379096 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:15.379096 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:15.379102 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:15.382273 (+ 3171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:15.382847 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:15.382848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:15.382849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:15.382908 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:15.383196 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:15.383197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:15.383197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:15.383342 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:15.383383 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:15.383773 (+ 390us) server_negotiation.cc:299] Negotiation successful
0926 20:49:15.383858 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 20:49:15.383963 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39349
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40969
--webserver_port=42093
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:40969,127.0.98.61:44089
--tserver_master_addrs=127.0.98.62:40969,127.0.98.61:44089,127.0.98.60:41871 with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:49:15.384196 7608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:15.372699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40969 (local address 127.0.98.2:34691)
0926 20:49:15.373159 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:15.373168 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:15.373478 (+ 310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:15.373993 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:15.373994 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:15.374249 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:15.374442 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:15.374446 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.375158 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.375159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:15.376038 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:15.376040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.376440 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.376441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:15.376563 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.377084 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:15.377096 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:15.378989 (+ 1893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:15.382369 (+ 3380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:15.382376 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:15.382379 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:15.382761 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:15.382978 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:15.382979 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:15.382979 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:15.383115 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:15.383426 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:15.383431 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:15.383600 (+ 169us) client_negotiation.cc:769] Sending connection context
0926 20:49:15.383715 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 20:49:15.383934 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":261,"thread_start_us":72,"threads_started":1}
I20260926 20:49:15.385421 7603 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40969
I20260926 20:49:15.385538 7603 heartbeater.cc:464] Registering TS with master...
I20260926 20:49:15.385733 7603 heartbeater.cc:511] Master 127.0.98.62:40969 requested a full tablet report, sending...
I20260926 20:49:15.386132 6496 ts_manager.cc:194] Re-registered known tserver with Master: 01cba43c9838497f8d037dec2c16e44a (127.0.98.2:44443)
I20260926 20:49:15.386656 6496 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34691
I20260926 20:49:15.387362 7471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:15.372758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49737 (local address 127.0.98.60:41871)
0926 20:49:15.372811 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:15.372814 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:15.373000 (+ 186us) server_negotiation.cc:407] Connection header received
0926 20:49:15.373602 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:15.373603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:15.373618 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:15.373649 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:15.374562 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.375305 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.376457 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.376773 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.376810 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.383107 (+ 6297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:15.383109 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:15.383109 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:15.383116 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:15.385967 (+ 2851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:15.386390 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:15.386391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:15.386392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:15.386455 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:15.386678 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:15.386679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:15.386679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:15.386815 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:15.386884 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:15.387126 (+ 242us) server_negotiation.cc:299] Negotiation successful
0926 20:49:15.387222 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 20:49:15.388273 7606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:15.372699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41871 (local address 127.0.98.2:49737)
0926 20:49:15.372877 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:15.372920 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:15.373480 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:15.373730 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:15.373735 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:15.374231 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:15.374443 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:15.374450 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.375407 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.375408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:15.376356 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:15.376358 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.376877 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.376877 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:15.376936 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.380641 (+ 3705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:15.380645 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:15.383005 (+ 2360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:15.386121 (+ 3116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:15.386122 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:15.386122 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:15.386314 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:15.386535 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:15.386536 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:15.386536 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:15.386596 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:15.386958 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:15.386959 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:15.387003 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 20:49:15.387079 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 20:49:15.387184 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":57,"thread_start_us":73,"threads_started":1}
I20260926 20:49:15.388315 7145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:15.372773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60879 (local address 127.0.98.61:44089)
0926 20:49:15.372817 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:15.372819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:15.373341 (+ 522us) server_negotiation.cc:407] Connection header received
0926 20:49:15.373602 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:15.373603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:15.373615 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:15.373649 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:15.374551 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.375312 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.376457 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.376748 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.376787 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.383042 (+ 6255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:15.383044 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:15.383044 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:15.383054 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:15.387046 (+ 3992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:15.387422 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:15.387423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:15.387424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:15.387491 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:15.387684 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:15.387686 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:15.387686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:15.387865 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:15.387940 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:15.388107 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 20:49:15.388203 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 20:49:15.388614 7602 heartbeater.cc:347] Connected to a master server at 127.0.98.60:41871
I20260926 20:49:15.388674 7602 heartbeater.cc:464] Registering TS with master...
I20260926 20:49:15.388645 7607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:15.372699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44089 (local address 127.0.98.2:60879)
0926 20:49:15.373275 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:15.373283 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:15.373491 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:15.373730 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:15.373735 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:15.374231 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:15.374443 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:15.374450 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.375407 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.375408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:15.376359 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:15.376361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.376866 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.376868 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:15.376936 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.380813 (+ 3877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:15.380817 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:15.382937 (+ 2120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:15.387123 (+ 4186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:15.387124 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:15.387124 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:15.387327 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:15.387558 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:15.387559 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:15.387559 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:15.387610 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:15.387939 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:15.387940 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:15.387977 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 20:49:15.388388 (+ 411us) client_negotiation.cc:240] Negotiation successful
0926 20:49:15.388502 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"mutex_wait_us":349,"thread_start_us":71,"threads_started":1}
I20260926 20:49:15.388757 7602 heartbeater.cc:511] Master 127.0.98.60:41871 requested a full tablet report, sending...
I20260926 20:49:15.388993 7604 heartbeater.cc:347] Connected to a master server at 127.0.98.61:44089
I20260926 20:49:15.389039 7604 heartbeater.cc:464] Registering TS with master...
I20260926 20:49:15.389086 7604 heartbeater.cc:511] Master 127.0.98.61:44089 requested a full tablet report, sending...
I20260926 20:49:15.389132 7286 ts_manager.cc:194] Registered new tserver with Master: 01cba43c9838497f8d037dec2c16e44a (127.0.98.2:44443)
I20260926 20:49:15.389389 7094 ts_manager.cc:194] Registered new tserver with Master: 01cba43c9838497f8d037dec2c16e44a (127.0.98.2:44443)
W20260926 20:49:15.467217 7609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:15.467459 7609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:15.467486 7609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:15.468688 7609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:15.468727 7609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:15.468753 7609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:49:15.470271 7609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39349
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42093
--tserver_master_addrs=127.0.98.62:40969,127.0.98.61:44089,127.0.98.60:41871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:15.470502 7609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:15.470691 7609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:49:15.475672 7609 server_base.cc:1034] running on GCE node
W20260926 20:49:15.475629 7614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:15.475629 7617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:15.475622 7615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:15.476083 7609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:15.476329 7609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:15.477473 7609 hybrid_clock.cc:648] HybridClock initialized: now 1790455755477452 us; error 40 us; skew 500 ppm
Sep 26 20:49:15 dist-test-slave-hwlj krb5kdc[6451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455755, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:49:15.480021 7609 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260926 20:49:15.480556 7609 webserver.cc:460] Webserver started at http://127.0.98.3:42093/ using document root <none> and password file <none>
I20260926 20:49:15.480801 7609 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:15.480854 7609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:15.482527 7609 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260926 20:49:15.483084 7624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:15.483290 7609 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:15.483374 7609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-2/wal
uuid: "345595a322944cad8c9ac66f823413a7"
format_stamp: "Formatted at 2026-09-26 20:49:14 on dist-test-slave-hwlj"
server_key: "78304a518f37b53d9e3f84f7399d2efa"
server_key_iv: "74bd3ba1e2a9aaa740cd4549dca69391"
server_key_version: "encryptionkey@0"
I20260926 20:49:15.483704 7609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:15.490402 7471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:15.488016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35714 (local address 127.0.98.60:41871)
0926 20:49:15.488077 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:15.488080 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:15.488089 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:49:15.488116 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:15.488117 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:15.488145 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:15.488218 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:49:15.488323 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.488802 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.489701 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.490081 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.490168 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.490234 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:15.490297 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 20:49:15.490334 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 20:49:15.490402 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:15.487948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41871 (local address 127.0.0.1:35714)
0926 20:49:15.488001 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:15.488012 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:15.488060 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:15.488222 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:15.488223 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:15.488224 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:49:15.488273 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:15.488275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.488884 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.488885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:15.489604 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:15.489605 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.490169 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.490170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:15.490230 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.490233 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:49:15.490291 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 20:49:15.490339 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 20:49:15.490700 7296 raft_consensus.cc:1203] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260926 20:49:15.490831 7325 sys_catalog.cc:455] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e93f52007174325a60947bfdcddc4d0. Latest consensus state: current_term: 1 leader_uuid: "4e93f52007174325a60947bfdcddc4d0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44089 } attrs { promote: false } } peers { permanent_uuid: "a19b05eba1c24e7db3c2ad66aa45fd33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41871 } attrs { promote: true } } }
I20260926 20:49:15.490943 7325 sys_catalog.cc:458] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [sys.catalog]: This master's current role is: LEARNER
I20260926 20:49:15.503345 7609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:15.505224 7609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:15.505339 7609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:15.505539 7609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:49:15.505784 7609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:49:15.505817 7609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:15.505838 7609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:49:15.505851 7609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:15.511811 7609 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39349
I20260926 20:49:15.511854 7737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39349 every 8 connection(s)
I20260926 20:49:15.512203 7609 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 20:49:15 dist-test-slave-hwlj krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455755, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 26 20:49:15 dist-test-slave-hwlj krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455755, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260926 20:49:15.519966 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 7609
Sep 26 20:49:15 dist-test-slave-hwlj krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455755, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 20:49:15.523734 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:15.512974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46935 (local address 127.0.98.62:40969)
0926 20:49:15.513124 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:15.513126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:15.513187 (+ 61us) server_negotiation.cc:407] Connection header received
0926 20:49:15.513895 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:15.513897 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:15.513909 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:15.513943 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:15.514926 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.515725 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.516694 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.516980 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.517055 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.519683 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:15.519684 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:15.519684 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:15.519691 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:15.521519 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:15.522006 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:15.522007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:15.522007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:15.522065 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:15.522715 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:15.522716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:15.522716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:15.522865 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:15.522927 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:15.523547 (+ 620us) server_negotiation.cc:299] Negotiation successful
0926 20:49:15.523627 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260926 20:49:15.523962 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:15.512860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40969 (local address 127.0.98.3:46935)
0926 20:49:15.513041 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:15.513085 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:15.513768 (+ 683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:15.514031 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:15.514036 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:15.514550 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:15.514817 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:15.514822 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.515900 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.515901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:15.516596 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:15.516598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.517073 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.517074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:15.517231 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.517996 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:15.518001 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:15.519585 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:15.521599 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:15.521605 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:15.521607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:15.521907 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:15.522479 (+ 572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:15.522480 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:15.522480 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:15.522622 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:15.522954 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:15.522959 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:15.523126 (+ 167us) client_negotiation.cc:769] Sending connection context
0926 20:49:15.523235 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 20:49:15.523407 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":3,"thread_start_us":92,"threads_started":1}
I20260926 20:49:15.524200 7471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:15.513028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60329 (local address 127.0.98.60:41871)
0926 20:49:15.513116 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:15.513119 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:15.513252 (+ 133us) server_negotiation.cc:407] Connection header received
0926 20:49:15.513894 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:15.513896 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:15.513912 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:15.513943 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:15.514925 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.515419 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.516864 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.517111 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.517188 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.521428 (+ 4240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:15.521429 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:15.521430 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:15.521436 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:15.523039 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:15.523408 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:15.523409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:15.523409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:15.523460 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:15.523652 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:15.523654 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:15.523654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:15.523797 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:15.523859 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:15.524043 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 20:49:15.524116 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 20:49:15.524232 7744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:15.512860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41871 (local address 127.0.98.3:60329)
0926 20:49:15.513193 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:15.513202 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:15.513768 (+ 566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:15.514043 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:15.514044 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:15.514553 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:15.514816 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:15.514822 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.515516 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.515517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:15.516753 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:15.516755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.517190 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.517192 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:15.517256 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.517907 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:15.517918 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:15.520713 (+ 2795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:15.523113 (+ 2400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:15.523114 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:15.523114 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:15.523333 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:15.523526 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:15.523527 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:15.523527 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:15.523584 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:15.523886 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:15.523887 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:15.523926 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 20:49:15.524018 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 20:49:15.524115 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":30,"thread_start_us":109,"threads_started":1}
I20260926 20:49:15.524410 7739 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40969
I20260926 20:49:15.524492 7739 heartbeater.cc:464] Registering TS with master...
I20260926 20:49:15.524634 7738 heartbeater.cc:347] Connected to a master server at 127.0.98.60:41871
I20260926 20:49:15.524679 7739 heartbeater.cc:511] Master 127.0.98.62:40969 requested a full tablet report, sending...
I20260926 20:49:15.524688 7738 heartbeater.cc:464] Registering TS with master...
I20260926 20:49:15.524760 7738 heartbeater.cc:511] Master 127.0.98.60:41871 requested a full tablet report, sending...
I20260926 20:49:15.525087 6496 ts_manager.cc:194] Re-registered known tserver with Master: 345595a322944cad8c9ac66f823413a7 (127.0.98.3:39349)
I20260926 20:49:15.525274 7286 ts_manager.cc:194] Registered new tserver with Master: 345595a322944cad8c9ac66f823413a7 (127.0.98.3:39349)
I20260926 20:49:15.525585 6496 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46935
I20260926 20:49:15.526031 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:15.512860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44089 (local address 127.0.98.3:53449)
0926 20:49:15.513262 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:15.513272 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:15.513779 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:15.514382 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:15.514383 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:15.514574 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:15.514816 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:15.514822 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.515806 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.515807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:15.516753 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:15.516755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.517267 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.517268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:15.517325 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.517927 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:15.517931 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:15.521012 (+ 3081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:15.523481 (+ 2469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:15.523482 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:15.523482 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:15.523700 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:15.524994 (+ 1294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:15.524995 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:15.524995 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:15.525059 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:15.525387 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:15.525388 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:15.525469 (+ 81us) client_negotiation.cc:769] Sending connection context
0926 20:49:15.525534 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 20:49:15.525627 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":236,"thread_start_us":58,"threads_started":1}
I20260926 20:49:15.526358 7145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:15.512920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53449 (local address 127.0.98.61:44089)
0926 20:49:15.513080 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:15.513082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:15.513345 (+ 263us) server_negotiation.cc:407] Connection header received
0926 20:49:15.513871 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:15.513872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:15.513884 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:15.513946 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:15.514959 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.515695 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.516841 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.517137 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.517216 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.521111 (+ 3895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:15.521113 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:15.521113 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:15.521122 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:15.523389 (+ 2267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:15.524845 (+ 1456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:15.524847 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:15.524847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:15.524914 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:15.525122 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:15.525123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:15.525124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:15.525249 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:15.525308 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:15.526183 (+ 875us) server_negotiation.cc:299] Negotiation successful
0926 20:49:15.526261 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260926 20:49:15.526949 7740 heartbeater.cc:347] Connected to a master server at 127.0.98.61:44089
I20260926 20:49:15.527005 7740 heartbeater.cc:464] Registering TS with master...
I20260926 20:49:15.527076 7740 heartbeater.cc:511] Master 127.0.98.61:44089 requested a full tablet report, sending...
I20260926 20:49:15.527403 7094 ts_manager.cc:194] Registered new tserver with Master: 345595a322944cad8c9ac66f823413a7 (127.0.98.3:39349)
I20260926 20:49:15.531226 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:15.521751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35720 (local address 127.0.98.60:41871)
0926 20:49:15.522201 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:15.522202 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:15.522533 (+ 331us) server_negotiation.cc:407] Connection header received
0926 20:49:15.522555 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:15.522556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:15.522568 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:15.522621 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:15.523445 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.524653 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.526085 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.526769 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.526836 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.528264 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:15.528265 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:15.528265 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:15.528272 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:15.530000 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:15.530364 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:15.530365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:15.530365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:15.530425 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:15.530633 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:15.530634 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:15.530635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:15.530752 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:15.530856 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:15.530988 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 20:49:15.531067 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":50,"threads_started":1}
I20260926 20:49:15.534149 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:15.521171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37154 (local address 127.0.98.61:44089)
0926 20:49:15.522402 (+ 1231us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:15.522404 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:15.522412 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:49:15.522431 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:15.522431 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:15.522443 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:15.522502 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:15.527803 (+ 5301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.528494 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.529481 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.529758 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.529844 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.531301 (+ 1457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:15.531302 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:15.531303 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:15.531310 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:15.533001 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:15.533347 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:15.533348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:15.533348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:15.533402 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:15.533601 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:15.533602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:15.533602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:15.533718 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:15.533814 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:15.533916 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 20:49:15.533992 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1208,"thread_start_us":55,"threads_started":1}
I20260926 20:49:15.534821 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:15.521070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43524 (local address 127.0.98.62:40969)
0926 20:49:15.521401 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:15.521403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:15.522064 (+ 661us) server_negotiation.cc:407] Connection header received
0926 20:49:15.522088 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:15.522088 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:15.522101 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:15.522195 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:15.528033 (+ 5838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.528706 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.529609 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.529953 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.530031 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.532012 (+ 1981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:15.532013 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:15.532013 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:15.532019 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:15.533531 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:15.533944 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:15.533945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:15.533945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:15.534007 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:15.534220 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:15.534221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:15.534221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:15.534377 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:15.534497 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:15.534619 (+ 122us) server_negotiation.cc:299] Negotiation successful
0926 20:49:15.534707 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":93,"threads_started":1}
I20260926 20:49:15.536648 6496 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43524:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:49:15.538185 6496 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:49:15.538921 7056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4e93f52007174325a60947bfdcddc4d0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44089 } attrs { promote: false } } peers { permanent_uuid: "a19b05eba1c24e7db3c2ad66aa45fd33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41871 } attrs { promote: true } } }
I20260926 20:49:15.539032 7056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0 [sys.catalog]: This master's current role is: LEADER
I20260926 20:49:15.539803 7056 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0: attempting to promote NON_VOTER a19b05eba1c24e7db3c2ad66aa45fd33 to VOTER
I20260926 20:49:15.539956 7056 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44089 } attrs { promote: false } } peers { permanent_uuid: "a19b05eba1c24e7db3c2ad66aa45fd33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41871 } attrs { promote: false } }
I20260926 20:49:15.540350 7104 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573 [term 1 FOLLOWER]: Refusing update from remote peer 4e93f52007174325a60947bfdcddc4d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 20:49:15.540364 7296 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [term 1 LEARNER]: Refusing update from remote peer 4e93f52007174325a60947bfdcddc4d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 20:49:15.540560 7756 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 20:49:15.540637 7054 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a19b05eba1c24e7db3c2ad66aa45fd33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41871 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 20:49:15.541385 7055 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, a19b05eba1c24e7db3c2ad66aa45fd33 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44089 } attrs { promote: false } } peers { permanent_uuid: "a19b05eba1c24e7db3c2ad66aa45fd33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41871 } attrs { promote: false } } }
I20260926 20:49:15.541532 7296 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, a19b05eba1c24e7db3c2ad66aa45fd33 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44089 } attrs { promote: false } } peers { permanent_uuid: "a19b05eba1c24e7db3c2ad66aa45fd33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41871 } attrs { promote: false } } }
I20260926 20:49:15.541560 7104 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, a19b05eba1c24e7db3c2ad66aa45fd33 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44089 } attrs { promote: false } } peers { permanent_uuid: "a19b05eba1c24e7db3c2ad66aa45fd33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41871 } attrs { promote: false } } }
I20260926 20:49:15.542289 7055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4e93f52007174325a60947bfdcddc4d0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44089 } attrs { promote: false } } peers { permanent_uuid: "a19b05eba1c24e7db3c2ad66aa45fd33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41871 } attrs { promote: false } } }
I20260926 20:49:15.542327 7056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4e93f52007174325a60947bfdcddc4d0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44089 } attrs { promote: false } } peers { permanent_uuid: "a19b05eba1c24e7db3c2ad66aa45fd33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41871 } attrs { promote: false } } }
I20260926 20:49:15.542385 7055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0 [sys.catalog]: This master's current role is: LEADER
I20260926 20:49:15.542399 7056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0 [sys.catalog]: This master's current role is: LEADER
I20260926 20:49:15.542459 7054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4e93f52007174325a60947bfdcddc4d0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44089 } attrs { promote: false } } peers { permanent_uuid: "a19b05eba1c24e7db3c2ad66aa45fd33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41871 } attrs { promote: false } } }
I20260926 20:49:15.542505 7054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0 [sys.catalog]: This master's current role is: LEADER
I20260926 20:49:15.542542 7250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4e93f52007174325a60947bfdcddc4d0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44089 } attrs { promote: false } } peers { permanent_uuid: "a19b05eba1c24e7db3c2ad66aa45fd33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41871 } attrs { promote: false } } }
I20260926 20:49:15.542632 7250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:49:15.542582 7325 sys_catalog.cc:455] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4e93f52007174325a60947bfdcddc4d0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44089 } attrs { promote: false } } peers { permanent_uuid: "a19b05eba1c24e7db3c2ad66aa45fd33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41871 } attrs { promote: false } } }
I20260926 20:49:15.542672 7325 sys_catalog.cc:458] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:49:15.548260 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:15.544777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53538 (local address 127.0.98.3:39349)
0926 20:49:15.544938 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:15.544943 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:15.544954 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:49:15.544993 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:15.544995 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:15.545013 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:15.545270 (+ 257us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:49:15.545329 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.546026 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.547139 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.547925 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.548017 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.548093 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:15.548147 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 20:49:15.548183 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":99,"threads_started":1}
I20260926 20:49:15.548251 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:15.544627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39349 (local address 127.0.0.1:53538)
0926 20:49:15.544758 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:15.544769 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:15.544820 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:15.545123 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:15.545124 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:15.545127 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:49:15.545194 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:15.545196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.546118 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.546119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:15.547049 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:15.547051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.548014 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.548015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:15.548071 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.548075 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:49:15.548130 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 20:49:15.548179 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":87,"threads_started":1}
I20260926 20:49:15.548436 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:15.544474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44443 (local address 127.0.0.1:50978)
0926 20:49:15.544512 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:15.544524 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:15.544563 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:15.545151 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:15.545152 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:15.545154 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:49:15.545201 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:15.545203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.546572 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.546573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:15.547437 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:15.547439 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.548244 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.548245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:15.548301 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.548305 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:49:15.548334 (+ 29us) client_negotiation.cc:240] Negotiation successful
0926 20:49:15.548371 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260926 20:49:15.548792 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:15.544537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46533 (local address 127.0.0.1:46614)
0926 20:49:15.544660 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:15.544670 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:15.544707 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:15.545349 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:15.545350 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:15.545352 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:49:15.545387 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:15.545388 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.546640 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.546641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:15.547672 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:15.547675 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.548578 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.548579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:15.548659 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.548664 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:49:15.548695 (+ 31us) client_negotiation.cc:240] Negotiation successful
0926 20:49:15.548727 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":86,"threads_started":1}
I20260926 20:49:15.548928 7672 tablet_service.cc:1467] Processing CreateTablet for tablet 07a8de6878574723841603586835ea85 (DEFAULT_TABLE table=test-table [id=e3ce4bcf54c64dfd87dfd69a55da1203]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:49:15.549222 7672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07a8de6878574723841603586835ea85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:49:15.549438 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:15.544972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46614 (local address 127.0.98.1:46533)
0926 20:49:15.545184 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:15.545188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:15.545197 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:49:15.545236 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:15.545238 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:15.545254 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:15.545473 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:49:15.545530 (+ 57us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.546560 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.547782 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.548466 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.548889 (+ 423us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.549283 (+ 394us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:15.549310 (+ 27us) server_negotiation.cc:299] Negotiation successful
0926 20:49:15.549361 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":79,"threads_started":1}
I20260926 20:49:15.549708 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:15.544782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50978 (local address 127.0.98.2:44443)
0926 20:49:15.544938 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:15.544942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:15.544953 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:49:15.545011 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:15.545012 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:15.545025 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:15.545279 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:49:15.545340 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.546466 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.547514 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.548148 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.549365 (+ 1217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.549449 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:15.549474 (+ 25us) server_negotiation.cc:299] Negotiation successful
0926 20:49:15.549655 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":97,"threads_started":1}
I20260926 20:49:15.550280 7536 tablet_service.cc:1467] Processing CreateTablet for tablet 07a8de6878574723841603586835ea85 (DEFAULT_TABLE table=test-table [id=e3ce4bcf54c64dfd87dfd69a55da1203]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:49:15.550573 7536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07a8de6878574723841603586835ea85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:49:15.552361 7399 tablet_service.cc:1467] Processing CreateTablet for tablet 07a8de6878574723841603586835ea85 (DEFAULT_TABLE table=test-table [id=e3ce4bcf54c64dfd87dfd69a55da1203]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:49:15.552686 7399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07a8de6878574723841603586835ea85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:49:15.552740 7768 tablet_bootstrap.cc:492] T 07a8de6878574723841603586835ea85 P 345595a322944cad8c9ac66f823413a7: Bootstrap starting.
I20260926 20:49:15.553377 7768 tablet_bootstrap.cc:654] T 07a8de6878574723841603586835ea85 P 345595a322944cad8c9ac66f823413a7: Neither blocks nor log segments found. Creating new log.
I20260926 20:49:15.553666 7768 log.cc:824] T 07a8de6878574723841603586835ea85 P 345595a322944cad8c9ac66f823413a7: Log is configured to *not* fsync() on all Append() calls
I20260926 20:49:15.554378 7768 tablet_bootstrap.cc:492] T 07a8de6878574723841603586835ea85 P 345595a322944cad8c9ac66f823413a7: No bootstrap required, opened a new log
I20260926 20:49:15.554440 7768 ts_tablet_manager.cc:1397] T 07a8de6878574723841603586835ea85 P 345595a322944cad8c9ac66f823413a7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:49:15.556003 7771 tablet_bootstrap.cc:492] T 07a8de6878574723841603586835ea85 P e728cd363cea4949a2a5d07380f215d8: Bootstrap starting.
I20260926 20:49:15.556702 7771 tablet_bootstrap.cc:654] T 07a8de6878574723841603586835ea85 P e728cd363cea4949a2a5d07380f215d8: Neither blocks nor log segments found. Creating new log.
I20260926 20:49:15.556983 7771 log.cc:824] T 07a8de6878574723841603586835ea85 P e728cd363cea4949a2a5d07380f215d8: Log is configured to *not* fsync() on all Append() calls
I20260926 20:49:15.557147 7768 raft_consensus.cc:348] T 07a8de6878574723841603586835ea85 P 345595a322944cad8c9ac66f823413a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01cba43c9838497f8d037dec2c16e44a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44443 } } peers { permanent_uuid: "345595a322944cad8c9ac66f823413a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39349 } } peers { permanent_uuid: "e728cd363cea4949a2a5d07380f215d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46533 } }
I20260926 20:49:15.557327 7768 raft_consensus.cc:374] T 07a8de6878574723841603586835ea85 P 345595a322944cad8c9ac66f823413a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:49:15.557372 7768 raft_consensus.cc:729] T 07a8de6878574723841603586835ea85 P 345595a322944cad8c9ac66f823413a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 345595a322944cad8c9ac66f823413a7, State: Initialized, Role: FOLLOWER
I20260926 20:49:15.557490 7768 consensus_queue.cc:260] T 07a8de6878574723841603586835ea85 P 345595a322944cad8c9ac66f823413a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01cba43c9838497f8d037dec2c16e44a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44443 } } peers { permanent_uuid: "345595a322944cad8c9ac66f823413a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39349 } } peers { permanent_uuid: "e728cd363cea4949a2a5d07380f215d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46533 } }
I20260926 20:49:15.557787 7739 heartbeater.cc:503] Master 127.0.98.62:40969 was elected leader, sending a full tablet report...
I20260926 20:49:15.557860 7771 tablet_bootstrap.cc:492] T 07a8de6878574723841603586835ea85 P e728cd363cea4949a2a5d07380f215d8: No bootstrap required, opened a new log
I20260926 20:49:15.557932 7771 ts_tablet_manager.cc:1397] T 07a8de6878574723841603586835ea85 P e728cd363cea4949a2a5d07380f215d8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:49:15.558679 7768 ts_tablet_manager.cc:1428] T 07a8de6878574723841603586835ea85 P 345595a322944cad8c9ac66f823413a7: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260926 20:49:15.559238 7769 tablet_bootstrap.cc:492] T 07a8de6878574723841603586835ea85 P 01cba43c9838497f8d037dec2c16e44a: Bootstrap starting.
I20260926 20:49:15.559648 7771 raft_consensus.cc:348] T 07a8de6878574723841603586835ea85 P e728cd363cea4949a2a5d07380f215d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01cba43c9838497f8d037dec2c16e44a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44443 } } peers { permanent_uuid: "345595a322944cad8c9ac66f823413a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39349 } } peers { permanent_uuid: "e728cd363cea4949a2a5d07380f215d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46533 } }
I20260926 20:49:15.559782 7769 tablet_bootstrap.cc:654] T 07a8de6878574723841603586835ea85 P 01cba43c9838497f8d037dec2c16e44a: Neither blocks nor log segments found. Creating new log.
I20260926 20:49:15.559824 7771 raft_consensus.cc:374] T 07a8de6878574723841603586835ea85 P e728cd363cea4949a2a5d07380f215d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:49:15.559867 7771 raft_consensus.cc:729] T 07a8de6878574723841603586835ea85 P e728cd363cea4949a2a5d07380f215d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e728cd363cea4949a2a5d07380f215d8, State: Initialized, Role: FOLLOWER
I20260926 20:49:15.559975 7771 consensus_queue.cc:260] T 07a8de6878574723841603586835ea85 P e728cd363cea4949a2a5d07380f215d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01cba43c9838497f8d037dec2c16e44a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44443 } } peers { permanent_uuid: "345595a322944cad8c9ac66f823413a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39349 } } peers { permanent_uuid: "e728cd363cea4949a2a5d07380f215d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46533 } }
I20260926 20:49:15.560067 7769 log.cc:824] T 07a8de6878574723841603586835ea85 P 01cba43c9838497f8d037dec2c16e44a: Log is configured to *not* fsync() on all Append() calls
I20260926 20:49:15.560225 7771 ts_tablet_manager.cc:1428] T 07a8de6878574723841603586835ea85 P e728cd363cea4949a2a5d07380f215d8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 20:49:15.560329 7466 heartbeater.cc:503] Master 127.0.98.62:40969 was elected leader, sending a full tablet report...
I20260926 20:49:15.560899 7769 tablet_bootstrap.cc:492] T 07a8de6878574723841603586835ea85 P 01cba43c9838497f8d037dec2c16e44a: No bootstrap required, opened a new log
I20260926 20:49:15.560967 7769 ts_tablet_manager.cc:1397] T 07a8de6878574723841603586835ea85 P 01cba43c9838497f8d037dec2c16e44a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:49:15.562446 7769 raft_consensus.cc:348] T 07a8de6878574723841603586835ea85 P 01cba43c9838497f8d037dec2c16e44a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01cba43c9838497f8d037dec2c16e44a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44443 } } peers { permanent_uuid: "345595a322944cad8c9ac66f823413a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39349 } } peers { permanent_uuid: "e728cd363cea4949a2a5d07380f215d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46533 } }
I20260926 20:49:15.562566 7769 raft_consensus.cc:374] T 07a8de6878574723841603586835ea85 P 01cba43c9838497f8d037dec2c16e44a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:49:15.562588 7769 raft_consensus.cc:729] T 07a8de6878574723841603586835ea85 P 01cba43c9838497f8d037dec2c16e44a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01cba43c9838497f8d037dec2c16e44a, State: Initialized, Role: FOLLOWER
I20260926 20:49:15.562714 7769 consensus_queue.cc:260] T 07a8de6878574723841603586835ea85 P 01cba43c9838497f8d037dec2c16e44a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01cba43c9838497f8d037dec2c16e44a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44443 } } peers { permanent_uuid: "345595a322944cad8c9ac66f823413a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39349 } } peers { permanent_uuid: "e728cd363cea4949a2a5d07380f215d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46533 } }
I20260926 20:49:15.562945 7769 ts_tablet_manager.cc:1428] T 07a8de6878574723841603586835ea85 P 01cba43c9838497f8d037dec2c16e44a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:49:15.562994 7603 heartbeater.cc:503] Master 127.0.98.62:40969 was elected leader, sending a full tablet report...
I20260926 20:49:15.568286 7776 raft_consensus.cc:482] T 07a8de6878574723841603586835ea85 P 01cba43c9838497f8d037dec2c16e44a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:49:15.568367 7776 raft_consensus.cc:504] T 07a8de6878574723841603586835ea85 P 01cba43c9838497f8d037dec2c16e44a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01cba43c9838497f8d037dec2c16e44a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44443 } } peers { permanent_uuid: "345595a322944cad8c9ac66f823413a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39349 } } peers { permanent_uuid: "e728cd363cea4949a2a5d07380f215d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46533 } }
I20260926 20:49:15.568629 7776 leader_election.cc:288] T 07a8de6878574723841603586835ea85 P 01cba43c9838497f8d037dec2c16e44a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 345595a322944cad8c9ac66f823413a7 (127.0.98.3:39349), e728cd363cea4949a2a5d07380f215d8 (127.0.98.1:46533)
I20260926 20:49:15.571678 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:15.568913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42935 (local address 127.0.98.3:39349)
0926 20:49:15.568978 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:15.568980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:15.568988 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:49:15.569010 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:15.569011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:15.569021 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:15.569074 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:49:15.569250 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.569741 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.570731 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.571302 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.571382 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.571392 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:15.571576 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 20:49:15.571621 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 20:49:15.571688 7607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:15.568742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39349 (local address 127.0.98.2:42935)
0926 20:49:15.568783 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:15.568794 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:15.568837 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:15.569086 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:15.569087 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:15.569088 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:49:15.569184 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:15.569185 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.569831 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.569832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:15.570586 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:15.570587 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.571380 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.571380 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:15.571478 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.571482 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:49:15.571556 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 20:49:15.571606 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260926 20:49:15.571705 7606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:15.568796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46533 (local address 127.0.98.2:38443)
0926 20:49:15.568843 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:15.568854 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:15.568888 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:15.569009 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:15.569010 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:15.569012 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:49:15.569081 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:15.569083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.569809 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.569810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:15.570784 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:15.570786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.571473 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.571473 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:15.571544 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.571547 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:49:15.571594 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 20:49:15.571621 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 20:49:15.571887 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:15.568833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38443 (local address 127.0.98.1:46533)
0926 20:49:15.568875 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:15.568877 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:15.568886 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:49:15.568940 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:15.568941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:15.568959 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:15.569012 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:49:15.569151 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.569736 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.570860 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.571401 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.571458 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.571466 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:15.571735 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 20:49:15.571804 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 20:49:15.572011 7692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a8de6878574723841603586835ea85" candidate_uuid: "01cba43c9838497f8d037dec2c16e44a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "345595a322944cad8c9ac66f823413a7" is_pre_election: true
I20260926 20:49:15.572139 7692 raft_consensus.cc:2454] T 07a8de6878574723841603586835ea85 P 345595a322944cad8c9ac66f823413a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01cba43c9838497f8d037dec2c16e44a in term 0.
I20260926 20:49:15.572345 7492 leader_election.cc:302] T 07a8de6878574723841603586835ea85 P 01cba43c9838497f8d037dec2c16e44a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01cba43c9838497f8d037dec2c16e44a, 345595a322944cad8c9ac66f823413a7; no voters:
I20260926 20:49:15.572376 7419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a8de6878574723841603586835ea85" candidate_uuid: "01cba43c9838497f8d037dec2c16e44a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e728cd363cea4949a2a5d07380f215d8" is_pre_election: true
I20260926 20:49:15.572474 7776 raft_consensus.cc:2781] T 07a8de6878574723841603586835ea85 P 01cba43c9838497f8d037dec2c16e44a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:49:15.572526 7776 raft_consensus.cc:482] T 07a8de6878574723841603586835ea85 P 01cba43c9838497f8d037dec2c16e44a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:49:15.572523 7419 raft_consensus.cc:2454] T 07a8de6878574723841603586835ea85 P e728cd363cea4949a2a5d07380f215d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01cba43c9838497f8d037dec2c16e44a in term 0.
I20260926 20:49:15.572557 7776 raft_consensus.cc:3037] T 07a8de6878574723841603586835ea85 P 01cba43c9838497f8d037dec2c16e44a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:49:15.573231 7776 raft_consensus.cc:504] T 07a8de6878574723841603586835ea85 P 01cba43c9838497f8d037dec2c16e44a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01cba43c9838497f8d037dec2c16e44a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44443 } } peers { permanent_uuid: "345595a322944cad8c9ac66f823413a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39349 } } peers { permanent_uuid: "e728cd363cea4949a2a5d07380f215d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46533 } }
I20260926 20:49:15.573366 7776 leader_election.cc:288] T 07a8de6878574723841603586835ea85 P 01cba43c9838497f8d037dec2c16e44a [CANDIDATE]: Term 1 election: Requested vote from peers 345595a322944cad8c9ac66f823413a7 (127.0.98.3:39349), e728cd363cea4949a2a5d07380f215d8 (127.0.98.1:46533)
I20260926 20:49:15.573485 7692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a8de6878574723841603586835ea85" candidate_uuid: "01cba43c9838497f8d037dec2c16e44a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "345595a322944cad8c9ac66f823413a7"
I20260926 20:49:15.573493 7419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a8de6878574723841603586835ea85" candidate_uuid: "01cba43c9838497f8d037dec2c16e44a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e728cd363cea4949a2a5d07380f215d8"
I20260926 20:49:15.573561 7419 raft_consensus.cc:3037] T 07a8de6878574723841603586835ea85 P e728cd363cea4949a2a5d07380f215d8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:49:15.573559 7692 raft_consensus.cc:3037] T 07a8de6878574723841603586835ea85 P 345595a322944cad8c9ac66f823413a7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:49:15.574218 7692 raft_consensus.cc:2454] T 07a8de6878574723841603586835ea85 P 345595a322944cad8c9ac66f823413a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01cba43c9838497f8d037dec2c16e44a in term 1.
I20260926 20:49:15.574334 7419 raft_consensus.cc:2454] T 07a8de6878574723841603586835ea85 P e728cd363cea4949a2a5d07380f215d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01cba43c9838497f8d037dec2c16e44a in term 1.
I20260926 20:49:15.574373 7492 leader_election.cc:302] T 07a8de6878574723841603586835ea85 P 01cba43c9838497f8d037dec2c16e44a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01cba43c9838497f8d037dec2c16e44a, 345595a322944cad8c9ac66f823413a7; no voters:
I20260926 20:49:15.574517 7776 raft_consensus.cc:2781] T 07a8de6878574723841603586835ea85 P 01cba43c9838497f8d037dec2c16e44a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:49:15.574734 7776 raft_consensus.cc:686] T 07a8de6878574723841603586835ea85 P 01cba43c9838497f8d037dec2c16e44a [term 1 LEADER]: Becoming Leader. State: Replica: 01cba43c9838497f8d037dec2c16e44a, State: Running, Role: LEADER
I20260926 20:49:15.574846 7776 consensus_queue.cc:237] T 07a8de6878574723841603586835ea85 P 01cba43c9838497f8d037dec2c16e44a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01cba43c9838497f8d037dec2c16e44a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44443 } } peers { permanent_uuid: "345595a322944cad8c9ac66f823413a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39349 } } peers { permanent_uuid: "e728cd363cea4949a2a5d07380f215d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46533 } }
I20260926 20:49:15.575585 6496 catalog_manager.cc:5495] T 07a8de6878574723841603586835ea85 P 01cba43c9838497f8d037dec2c16e44a reported cstate change: term changed from 0 to 1, leader changed from <none> to 01cba43c9838497f8d037dec2c16e44a (127.0.98.2). New cstate: current_term: 1 leader_uuid: "01cba43c9838497f8d037dec2c16e44a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01cba43c9838497f8d037dec2c16e44a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44443 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "345595a322944cad8c9ac66f823413a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e728cd363cea4949a2a5d07380f215d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46533 } health_report { overall_health: UNKNOWN } } }
I20260926 20:49:15.586395 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:15.584140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50986 (local address 127.0.98.2:44443)
0926 20:49:15.584178 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:15.584180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:15.584190 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:49:15.584258 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:15.584259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:15.584280 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:15.584331 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:49:15.584448 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.584891 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.585621 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.585805 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.585860 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.586037 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:49:15.586151 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:49:15.586243 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:15.586295 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 20:49:15.586315 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 20:49:15.588208 7419 raft_consensus.cc:1261] T 07a8de6878574723841603586835ea85 P e728cd363cea4949a2a5d07380f215d8 [term 1 FOLLOWER]: Refusing update from remote peer 01cba43c9838497f8d037dec2c16e44a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:49:15.588280 7692 raft_consensus.cc:1261] T 07a8de6878574723841603586835ea85 P 345595a322944cad8c9ac66f823413a7 [term 1 FOLLOWER]: Refusing update from remote peer 01cba43c9838497f8d037dec2c16e44a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:49:15.588574 7776 consensus_queue.cc:1035] T 07a8de6878574723841603586835ea85 P 01cba43c9838497f8d037dec2c16e44a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e728cd363cea4949a2a5d07380f215d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:49:15.588718 7777 consensus_queue.cc:1035] T 07a8de6878574723841603586835ea85 P 01cba43c9838497f8d037dec2c16e44a [LEADER]: Connected to new peer: Peer: permanent_uuid: "345595a322944cad8c9ac66f823413a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:49:15.591962 7780 mvcc.cc:205] Tried to move back new op lower bound from 7333706774886395904 to 7333706774835298304. Current Snapshot: MvccSnapshot[applied={T|T < 7333706774886395904}]
I20260926 20:49:15.592300 7781 mvcc.cc:205] Tried to move back new op lower bound from 7333706774886395904 to 7333706774835298304. Current Snapshot: MvccSnapshot[applied={T|T < 7333706774886395904}]
I20260926 20:49:15.595923 6496 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:49:15.596022 6496 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:49:15.597285 6496 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4e93f52007174325a60947bfdcddc4d0: Sending DeleteTablet for 3 replicas of tablet 07a8de6878574723841603586835ea85
I20260926 20:49:15.597600 7399 tablet_service.cc:1514] Processing DeleteTablet for tablet 07a8de6878574723841603586835ea85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:49:15 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46614
I20260926 20:49:15.597714 7536 tablet_service.cc:1514] Processing DeleteTablet for tablet 07a8de6878574723841603586835ea85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:49:15 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50978
I20260926 20:49:15.597832 7672 tablet_service.cc:1514] Processing DeleteTablet for tablet 07a8de6878574723841603586835ea85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:49:15 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53538
I20260926 20:49:15.597944 7791 tablet_replica.cc:330] T 07a8de6878574723841603586835ea85 P 01cba43c9838497f8d037dec2c16e44a: stopping tablet replica
I20260926 20:49:15.598039 7791 raft_consensus.cc:2229] T 07a8de6878574723841603586835ea85 P 01cba43c9838497f8d037dec2c16e44a [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:49:15.598169 7791 raft_consensus.cc:2258] T 07a8de6878574723841603586835ea85 P 01cba43c9838497f8d037dec2c16e44a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:49:15.598696 7791 ts_tablet_manager.cc:1905] T 07a8de6878574723841603586835ea85 P 01cba43c9838497f8d037dec2c16e44a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:49:15.598985 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 6464
I20260926 20:49:15.601970 7791 ts_tablet_manager.cc:1918] T 07a8de6878574723841603586835ea85 P 01cba43c9838497f8d037dec2c16e44a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:49:15.602041 7791 log.cc:1196] T 07a8de6878574723841603586835ea85 P 01cba43c9838497f8d037dec2c16e44a: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-1/wal/wals/07a8de6878574723841603586835ea85
I20260926 20:49:15.602303 7791 ts_tablet_manager.cc:1939] T 07a8de6878574723841603586835ea85 P 01cba43c9838497f8d037dec2c16e44a: Deleting consensus metadata
I20260926 20:49:15.603878 7792 tablet_replica.cc:330] T 07a8de6878574723841603586835ea85 P 345595a322944cad8c9ac66f823413a7: stopping tablet replica
I20260926 20:49:15.604074 7790 tablet_replica.cc:330] T 07a8de6878574723841603586835ea85 P e728cd363cea4949a2a5d07380f215d8: stopping tablet replica
I20260926 20:49:15.604234 7790 raft_consensus.cc:2229] T 07a8de6878574723841603586835ea85 P e728cd363cea4949a2a5d07380f215d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:49:15.604395 7792 raft_consensus.cc:2229] T 07a8de6878574723841603586835ea85 P 345595a322944cad8c9ac66f823413a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:49:15.604584 7790 raft_consensus.cc:2258] T 07a8de6878574723841603586835ea85 P e728cd363cea4949a2a5d07380f215d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:49:15.604748 7792 raft_consensus.cc:2258] T 07a8de6878574723841603586835ea85 P 345595a322944cad8c9ac66f823413a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:49:15.605366 7792 ts_tablet_manager.cc:1905] T 07a8de6878574723841603586835ea85 P 345595a322944cad8c9ac66f823413a7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 20:49:15.605532 7490 connection.cc:535] server connection from 127.0.0.1:50978 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 20:49:15.606660 7790 ts_tablet_manager.cc:1905] T 07a8de6878574723841603586835ea85 P e728cd363cea4949a2a5d07380f215d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:49:15.608661 7790 ts_tablet_manager.cc:1918] T 07a8de6878574723841603586835ea85 P e728cd363cea4949a2a5d07380f215d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:49:15.608744 7790 log.cc:1196] T 07a8de6878574723841603586835ea85 P e728cd363cea4949a2a5d07380f215d8: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-0/wal/wals/07a8de6878574723841603586835ea85
I20260926 20:49:15.609010 7790 ts_tablet_manager.cc:1939] T 07a8de6878574723841603586835ea85 P e728cd363cea4949a2a5d07380f215d8: Deleting consensus metadata
W20260926 20:49:15.609356 7354 connection.cc:441] server connection from 127.0.0.1:46614 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46614 (request call id 1) could send its response
I20260926 20:49:15.610069 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:15.598526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35726 (local address 127.0.98.60:41871)
0926 20:49:15.598572 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:15.598574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:15.601138 (+ 2564us) server_negotiation.cc:407] Connection header received
0926 20:49:15.601213 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:15.601214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:15.601227 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:15.601276 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:15.601881 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.602642 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.603370 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.603589 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.603676 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.605965 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:15.605967 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:15.605967 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:15.605974 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:15.607590 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:15.609161 (+ 1571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:15.609162 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:15.609163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:15.609220 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:15.609487 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:15.609488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:15.609488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:15.609609 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:15.609671 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:15.609882 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 20:49:15.609962 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 20:49:15.611867 7792 ts_tablet_manager.cc:1918] T 07a8de6878574723841603586835ea85 P 345595a322944cad8c9ac66f823413a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:49:15.611948 7792 log.cc:1196] T 07a8de6878574723841603586835ea85 P 345595a322944cad8c9ac66f823413a7: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-2/wal/wals/07a8de6878574723841603586835ea85
I20260926 20:49:15.612249 7792 ts_tablet_manager.cc:1939] T 07a8de6878574723841603586835ea85 P 345595a322944cad8c9ac66f823413a7: Deleting consensus metadata
W20260926 20:49:15.612612 7625 connection.cc:441] server connection from 127.0.0.1:53538 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53538 (request call id 1) could send its response
I20260926 20:49:15.615207 7471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:15.607398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35742 (local address 127.0.98.60:41871)
0926 20:49:15.607439 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:15.607441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:15.608005 (+ 564us) server_negotiation.cc:407] Connection header received
0926 20:49:15.608099 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:15.608100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:15.608110 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:15.608138 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:15.608775 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.609479 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.610641 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.610816 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.610909 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.612327 (+ 1418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:15.612328 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:15.612329 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:15.612335 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:15.613897 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:15.614286 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:15.614288 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:15.614288 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:15.614353 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:15.614576 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:15.614578 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:15.614578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:15.614710 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:15.614782 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:15.615014 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 20:49:15.615104 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 20:49:15.618062 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:15.607709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37156 (local address 127.0.98.61:44089)
0926 20:49:15.607771 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:15.607773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:15.610959 (+ 3186us) server_negotiation.cc:407] Connection header received
0926 20:49:15.611029 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:15.611030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:15.611043 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:15.611349 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:15.611920 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.612468 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.613557 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:15.613778 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:15.613883 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:15.615190 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:15.615191 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:15.615191 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:15.615199 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:15.617028 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:15.617343 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:15.617344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:15.617345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:15.617402 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:15.617593 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:15.617594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:15.617594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:15.617726 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:15.617818 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:15.617892 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 20:49:15.617972 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 20:49:16.593513 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:16.593180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40969 (local address 127.0.98.2:35755)
0926 20:49:16.593340 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:16.593432 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":107,"threads_started":1}
I20260926 20:49:16.593932 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:16.593626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40969 (local address 127.0.98.1:37955)
0926 20:49:16.593736 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:16.593822 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":48,"threads_started":1}
W20260926 20:49:16.594138 7466 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40969 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:40969: connect: Connection refused (error 111)
W20260926 20:49:16.594230 7603 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40969 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:40969: connect: Connection refused (error 111)
I20260926 20:49:16.594411 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:16.594069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40969 (local address 127.0.98.3:58093)
0926 20:49:16.594231 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:16.594369 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":44,"threads_started":1}
W20260926 20:49:16.594686 7739 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40969 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:40969: connect: Connection refused (error 111)
I20260926 20:49:17.097915 7804 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4e93f52007174325a60947bfdcddc4d0)
I20260926 20:49:17.098133 7804 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44089 } attrs { promote: false } } peers { permanent_uuid: "a19b05eba1c24e7db3c2ad66aa45fd33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41871 } attrs { promote: false } }
I20260926 20:49:17.098213 7805 raft_consensus.cc:482] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4e93f52007174325a60947bfdcddc4d0)
I20260926 20:49:17.098340 7805 raft_consensus.cc:504] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44089 } attrs { promote: false } } peers { permanent_uuid: "a19b05eba1c24e7db3c2ad66aa45fd33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41871 } attrs { promote: false } }
I20260926 20:49:17.098452 7804 leader_election.cc:288] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4e93f52007174325a60947bfdcddc4d0 (127.0.98.62:40969), a19b05eba1c24e7db3c2ad66aa45fd33 (127.0.98.60:41871)
I20260926 20:49:17.098579 7805 leader_election.cc:288] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4e93f52007174325a60947bfdcddc4d0 (127.0.98.62:40969), 27e19353428e42f09a3f45b96fb8a573 (127.0.98.61:44089)
I20260926 20:49:17.098876 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:17.098637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40969 (local address 127.0.0.1:43906)
0926 20:49:17.098766 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:17.098831 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":52,"threads_started":1}
W20260926 20:49:17.099020 7078 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40969: connect: Connection refused (error 111)
I20260926 20:49:17.099687 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:17.098809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40969 (local address 127.0.0.1:43920)
0926 20:49:17.099603 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:17.099652 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":784,"thread_start_us":40,"threads_started":1}
I20260926 20:49:17.099704 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:17.099606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40969 (local address 127.0.0.1:43942)
0926 20:49:17.099646 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:17.099672 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260926 20:49:17.099817 7270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40969: connect: Connection refused (error 111)
W20260926 20:49:17.099984 7078 leader_election.cc:334] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4e93f52007174325a60947bfdcddc4d0 (127.0.98.62:40969): Network error: Client connection negotiation failed: client connection to 127.0.98.62:40969: connect: Connection refused (error 111)
I20260926 20:49:17.100452 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:17.100333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40969 (local address 127.0.0.1:43946)
0926 20:49:17.100394 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:17.100425 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 20:49:17.101449 7270 leader_election.cc:334] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4e93f52007174325a60947bfdcddc4d0 (127.0.98.62:40969): Network error: Client connection negotiation failed: client connection to 127.0.98.62:40969: connect: Connection refused (error 111)
I20260926 20:49:17.102212 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:17.098883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37172 (local address 127.0.98.61:44089)
0926 20:49:17.099131 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:17.099133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:17.099139 (+ 6us) server_negotiation.cc:407] Connection header received
0926 20:49:17.099161 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:17.099162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:17.099171 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:17.099236 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:49:17.099431 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:17.100358 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:17.101446 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:17.101913 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:17.101964 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:17.101973 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:17.102143 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 20:49:17.102171 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":56,"threads_started":1}
I20260926 20:49:17.102241 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:17.098681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44089 (local address 127.0.0.1:37172)
0926 20:49:17.098946 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:17.098957 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:17.099010 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:17.099249 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:17.099250 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:17.099252 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:49:17.099310 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:17.099313 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:17.100429 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:17.100431 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:17.101353 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:17.101355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:17.101996 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:17.101997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:17.102063 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:17.102068 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:49:17.102135 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 20:49:17.102174 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":38,"threads_started":1}
I20260926 20:49:17.102475 7104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a19b05eba1c24e7db3c2ad66aa45fd33" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "27e19353428e42f09a3f45b96fb8a573" is_pre_election: true
I20260926 20:49:17.102612 7104 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a19b05eba1c24e7db3c2ad66aa45fd33 in term 1.
I20260926 20:49:17.102707 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:17.098654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41871 (local address 127.0.0.1:35748)
0926 20:49:17.098939 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:17.098950 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:17.099010 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:17.099556 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:17.099556 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:17.099559 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:49:17.099594 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:17.099595 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:17.100803 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:17.100804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:17.101648 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:17.101649 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:17.102445 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:17.102446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:17.102513 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:17.102517 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:49:17.102602 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 20:49:17.102645 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":49,"threads_started":1}
I20260926 20:49:17.102718 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:17.098864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35748 (local address 127.0.98.60:41871)
0926 20:49:17.099451 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:17.099454 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:17.099459 (+ 5us) server_negotiation.cc:407] Connection header received
0926 20:49:17.099478 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:17.099478 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:17.099487 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:17.099561 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:49:17.100023 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:17.100725 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:17.101744 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:17.102275 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:17.102441 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:17.102453 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:17.102623 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 20:49:17.102665 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":41,"threads_started":1}
I20260926 20:49:17.102761 7272 leader_election.cc:302] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27e19353428e42f09a3f45b96fb8a573, a19b05eba1c24e7db3c2ad66aa45fd33; no voters: 4e93f52007174325a60947bfdcddc4d0
I20260926 20:49:17.102941 7805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 20:49:17.102988 7805 raft_consensus.cc:482] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4e93f52007174325a60947bfdcddc4d0)
I20260926 20:49:17.103040 7805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [term 1 FOLLOWER]: Advancing to term 2
I20260926 20:49:17.102988 7296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "27e19353428e42f09a3f45b96fb8a573" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a19b05eba1c24e7db3c2ad66aa45fd33" is_pre_election: true
I20260926 20:49:17.103734 7805 raft_consensus.cc:504] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44089 } attrs { promote: false } } peers { permanent_uuid: "a19b05eba1c24e7db3c2ad66aa45fd33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41871 } attrs { promote: false } }
I20260926 20:49:17.103967 7296 raft_consensus.cc:2379] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27e19353428e42f09a3f45b96fb8a573 in current term 2: Already voted for candidate a19b05eba1c24e7db3c2ad66aa45fd33 in this term.
I20260926 20:49:17.104019 7805 leader_election.cc:288] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [CANDIDATE]: Term 2 election: Requested vote from peers 4e93f52007174325a60947bfdcddc4d0 (127.0.98.62:40969), 27e19353428e42f09a3f45b96fb8a573 (127.0.98.61:44089)
I20260926 20:49:17.104089 7104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a19b05eba1c24e7db3c2ad66aa45fd33" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "27e19353428e42f09a3f45b96fb8a573"
I20260926 20:49:17.104136 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:17.104022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40969 (local address 127.0.0.1:43948)
0926 20:49:17.104091 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:17.104116 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 20:49:17.104163 7104 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573 [term 1 FOLLOWER]: Advancing to term 2
I20260926 20:49:17.104172 7081 leader_election.cc:302] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27e19353428e42f09a3f45b96fb8a573; no voters: 4e93f52007174325a60947bfdcddc4d0, a19b05eba1c24e7db3c2ad66aa45fd33
I20260926 20:49:17.104419 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:17.104345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40969 (local address 127.0.0.1:43960)
0926 20:49:17.104383 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:17.104406 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260926 20:49:17.104517 7270 leader_election.cc:334] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4e93f52007174325a60947bfdcddc4d0 (127.0.98.62:40969): Network error: Client connection negotiation failed: client connection to 127.0.98.62:40969: connect: Connection refused (error 111)
I20260926 20:49:17.104894 7104 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a19b05eba1c24e7db3c2ad66aa45fd33 in term 2.
I20260926 20:49:17.105005 7804 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 20:49:17.105103 7272 leader_election.cc:302] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27e19353428e42f09a3f45b96fb8a573, a19b05eba1c24e7db3c2ad66aa45fd33; no voters: 4e93f52007174325a60947bfdcddc4d0
I20260926 20:49:17.105195 7805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 20:49:17.105316 7805 raft_consensus.cc:686] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [term 2 LEADER]: Becoming Leader. State: Replica: a19b05eba1c24e7db3c2ad66aa45fd33, State: Running, Role: LEADER
I20260926 20:49:17.105397 7805 consensus_queue.cc:237] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44089 } attrs { promote: false } } peers { permanent_uuid: "a19b05eba1c24e7db3c2ad66aa45fd33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41871 } attrs { promote: false } }
I20260926 20:49:17.105746 7814 sys_catalog.cc:455] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a19b05eba1c24e7db3c2ad66aa45fd33. Latest consensus state: current_term: 2 leader_uuid: "a19b05eba1c24e7db3c2ad66aa45fd33" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44089 } attrs { promote: false } } peers { permanent_uuid: "a19b05eba1c24e7db3c2ad66aa45fd33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41871 } attrs { promote: false } } }
I20260926 20:49:17.105835 7814 sys_catalog.cc:458] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [sys.catalog]: This master's current role is: LEADER
I20260926 20:49:17.105996 7816 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:49:17.106501 7816 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:49:17.106582 7816 catalog_manager.cc:1259] Loaded cluster ID: d234dc66a6104be1b78c57501e0b0d6d
I20260926 20:49:17.106618 7816 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:49:17.106787 7816 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:49:17.106866 7816 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33: Loaded TSK: 0
I20260926 20:49:17.107034 7816 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:49:17.155067 7285 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35742:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:49:17.156795 7285 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:49:17.157796 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:17.157691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40969 (local address 127.0.0.1:43968)
0926 20:49:17.157739 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:17.157771 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 20:49:17.157832 7104 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573 [term 2 FOLLOWER]: Refusing update from remote peer a19b05eba1c24e7db3c2ad66aa45fd33: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260926 20:49:17.158105 7805 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260926 20:49:17.158170 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:17.158053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40969 (local address 127.0.0.1:43976)
0926 20:49:17.158126 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:17.158153 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 20:49:17.158277 7270 consensus_peers.cc:487] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 -> Peer 4e93f52007174325a60947bfdcddc4d0 (127.0.98.62:40969): Couldn't send request to peer 4e93f52007174325a60947bfdcddc4d0. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40969: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 20:49:17.158710 7804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a19b05eba1c24e7db3c2ad66aa45fd33. Latest consensus state: current_term: 2 leader_uuid: "a19b05eba1c24e7db3c2ad66aa45fd33" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44089 } attrs { promote: false } } peers { permanent_uuid: "a19b05eba1c24e7db3c2ad66aa45fd33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41871 } attrs { promote: false } } }
I20260926 20:49:17.158812 7804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:49:17.159219 7814 sys_catalog.cc:455] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "a19b05eba1c24e7db3c2ad66aa45fd33" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44089 } attrs { promote: false } } peers { permanent_uuid: "a19b05eba1c24e7db3c2ad66aa45fd33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41871 } attrs { promote: false } } }
I20260926 20:49:17.159303 7814 sys_catalog.cc:458] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33 [sys.catalog]: This master's current role is: LEADER
I20260926 20:49:17.159420 7804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "a19b05eba1c24e7db3c2ad66aa45fd33" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e93f52007174325a60947bfdcddc4d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "27e19353428e42f09a3f45b96fb8a573" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44089 } attrs { promote: false } } peers { permanent_uuid: "a19b05eba1c24e7db3c2ad66aa45fd33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41871 } attrs { promote: false } } }
I20260926 20:49:17.159476 7818 mvcc.cc:205] Tried to move back new op lower bound from 7333706781316382720 to 7333706781104271360. Current Snapshot: MvccSnapshot[applied={T|T < 7333706781316382720}]
I20260926 20:49:17.159500 7804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27e19353428e42f09a3f45b96fb8a573 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:49:17.165225 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:17.161318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44443 (local address 127.0.0.1:51002)
0926 20:49:17.161371 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:17.161382 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:17.161429 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:17.161765 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:17.161766 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:17.161768 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:49:17.161823 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:17.161824 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:17.162951 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:17.162951 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:17.164211 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:17.164213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:17.164956 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:17.164957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:17.165005 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:17.165010 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:49:17.165113 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 20:49:17.165157 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":6}
I20260926 20:49:17.165239 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:17.161507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51002 (local address 127.0.98.2:44443)
0926 20:49:17.161614 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:17.161616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:17.161625 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:49:17.161652 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:17.161653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:17.161666 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:17.161887 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:49:17.161929 (+ 42us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:17.162720 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:17.164308 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:17.164867 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:17.164952 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:17.164966 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:17.165124 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 20:49:17.165174 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":64,"threads_started":1}
I20260926 20:49:17.165812 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:17.161431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39349 (local address 127.0.0.1:53552)
0926 20:49:17.161476 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:17.161493 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:17.161546 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:17.161920 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:17.161921 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:17.161924 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:49:17.161978 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:17.161980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:17.162855 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:17.162856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:17.164211 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:17.164213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:17.165605 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:17.165606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:17.165673 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:17.165677 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:49:17.165706 (+ 29us) client_negotiation.cc:240] Negotiation successful
0926 20:49:17.165743 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19,"mutex_wait_us":3}
I20260926 20:49:17.166105 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:17.161497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53552 (local address 127.0.98.3:39349)
0926 20:49:17.161741 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:17.161743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:17.161751 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:49:17.161776 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:17.161776 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:17.161786 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:17.161875 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:49:17.162054 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:17.162765 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:17.165004 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:17.165503 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:17.165949 (+ 446us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:17.165965 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:17.165986 (+ 21us) server_negotiation.cc:299] Negotiation successful
0926 20:49:17.166030 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":48,"threads_started":1}
I20260926 20:49:17.166687 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:17.161629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46620 (local address 127.0.98.1:46533)
0926 20:49:17.161837 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:17.161840 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:17.162900 (+ 1060us) server_negotiation.cc:407] Connection header received
0926 20:49:17.162969 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:17.162970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:17.162979 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:17.163031 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:49:17.163131 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:17.163601 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:17.165698 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:17.166337 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:17.166414 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:17.166425 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:17.166573 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 20:49:17.166616 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":54,"threads_started":1}
I20260926 20:49:17.166687 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:17.161326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46533 (local address 127.0.0.1:46620)
0926 20:49:17.162849 (+ 1523us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:17.162859 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:17.162913 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:17.163041 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:17.163042 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:17.163043 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:49:17.163083 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:17.163084 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:17.163688 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:17.163689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:17.164851 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:17.164854 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:17.166420 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:17.166421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:17.166482 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:17.166486 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:49:17.166564 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 20:49:17.166624 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1501,"mutex_wait_us":134,"thread_start_us":122,"threads_started":1}
I20260926 20:49:17.166862 7536 tablet_service.cc:1467] Processing CreateTablet for tablet cc664b9806254d9abd018203d5a42ab7 (DEFAULT_TABLE table=test-table [id=5255b402fb624242873bd0c452dcfeda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:49:17.166990 7536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc664b9806254d9abd018203d5a42ab7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:49:17.167183 7399 tablet_service.cc:1467] Processing CreateTablet for tablet cc664b9806254d9abd018203d5a42ab7 (DEFAULT_TABLE table=test-table [id=5255b402fb624242873bd0c452dcfeda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:49:17.167325 7399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc664b9806254d9abd018203d5a42ab7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:49:17.168394 7826 tablet_bootstrap.cc:492] T cc664b9806254d9abd018203d5a42ab7 P 01cba43c9838497f8d037dec2c16e44a: Bootstrap starting.
I20260926 20:49:17.168449 7827 tablet_bootstrap.cc:492] T cc664b9806254d9abd018203d5a42ab7 P e728cd363cea4949a2a5d07380f215d8: Bootstrap starting.
I20260926 20:49:17.168640 7672 tablet_service.cc:1467] Processing CreateTablet for tablet cc664b9806254d9abd018203d5a42ab7 (DEFAULT_TABLE table=test-table [id=5255b402fb624242873bd0c452dcfeda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:49:17.168764 7672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc664b9806254d9abd018203d5a42ab7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:49:17.168803 7827 tablet_bootstrap.cc:654] T cc664b9806254d9abd018203d5a42ab7 P e728cd363cea4949a2a5d07380f215d8: Neither blocks nor log segments found. Creating new log.
I20260926 20:49:17.169117 7826 tablet_bootstrap.cc:654] T cc664b9806254d9abd018203d5a42ab7 P 01cba43c9838497f8d037dec2c16e44a: Neither blocks nor log segments found. Creating new log.
I20260926 20:49:17.169651 7826 tablet_bootstrap.cc:492] T cc664b9806254d9abd018203d5a42ab7 P 01cba43c9838497f8d037dec2c16e44a: No bootstrap required, opened a new log
I20260926 20:49:17.169708 7826 ts_tablet_manager.cc:1397] T cc664b9806254d9abd018203d5a42ab7 P 01cba43c9838497f8d037dec2c16e44a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:49:17.169817 7826 raft_consensus.cc:348] T cc664b9806254d9abd018203d5a42ab7 P 01cba43c9838497f8d037dec2c16e44a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01cba43c9838497f8d037dec2c16e44a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44443 } } peers { permanent_uuid: "345595a322944cad8c9ac66f823413a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39349 } } peers { permanent_uuid: "e728cd363cea4949a2a5d07380f215d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46533 } }
I20260926 20:49:17.169886 7826 raft_consensus.cc:374] T cc664b9806254d9abd018203d5a42ab7 P 01cba43c9838497f8d037dec2c16e44a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:49:17.169905 7826 raft_consensus.cc:729] T cc664b9806254d9abd018203d5a42ab7 P 01cba43c9838497f8d037dec2c16e44a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01cba43c9838497f8d037dec2c16e44a, State: Initialized, Role: FOLLOWER
I20260926 20:49:17.169957 7826 consensus_queue.cc:260] T cc664b9806254d9abd018203d5a42ab7 P 01cba43c9838497f8d037dec2c16e44a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01cba43c9838497f8d037dec2c16e44a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44443 } } peers { permanent_uuid: "345595a322944cad8c9ac66f823413a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39349 } } peers { permanent_uuid: "e728cd363cea4949a2a5d07380f215d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46533 } }
I20260926 20:49:17.170325 7831 tablet_bootstrap.cc:492] T cc664b9806254d9abd018203d5a42ab7 P 345595a322944cad8c9ac66f823413a7: Bootstrap starting.
I20260926 20:49:17.170703 7831 tablet_bootstrap.cc:654] T cc664b9806254d9abd018203d5a42ab7 P 345595a322944cad8c9ac66f823413a7: Neither blocks nor log segments found. Creating new log.
I20260926 20:49:17.170105 7826 ts_tablet_manager.cc:1428] T cc664b9806254d9abd018203d5a42ab7 P 01cba43c9838497f8d037dec2c16e44a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 20:49:17.170768 7827 tablet_bootstrap.cc:492] T cc664b9806254d9abd018203d5a42ab7 P e728cd363cea4949a2a5d07380f215d8: No bootstrap required, opened a new log
I20260926 20:49:17.170812 7827 ts_tablet_manager.cc:1397] T cc664b9806254d9abd018203d5a42ab7 P e728cd363cea4949a2a5d07380f215d8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:49:17.170855 7602 heartbeater.cc:503] Master 127.0.98.60:41871 was elected leader, sending a full tablet report...
I20260926 20:49:17.170958 7827 raft_consensus.cc:348] T cc664b9806254d9abd018203d5a42ab7 P e728cd363cea4949a2a5d07380f215d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01cba43c9838497f8d037dec2c16e44a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44443 } } peers { permanent_uuid: "345595a322944cad8c9ac66f823413a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39349 } } peers { permanent_uuid: "e728cd363cea4949a2a5d07380f215d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46533 } }
I20260926 20:49:17.171032 7827 raft_consensus.cc:374] T cc664b9806254d9abd018203d5a42ab7 P e728cd363cea4949a2a5d07380f215d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:49:17.171054 7827 raft_consensus.cc:729] T cc664b9806254d9abd018203d5a42ab7 P e728cd363cea4949a2a5d07380f215d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e728cd363cea4949a2a5d07380f215d8, State: Initialized, Role: FOLLOWER
I20260926 20:49:17.171111 7827 consensus_queue.cc:260] T cc664b9806254d9abd018203d5a42ab7 P e728cd363cea4949a2a5d07380f215d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01cba43c9838497f8d037dec2c16e44a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44443 } } peers { permanent_uuid: "345595a322944cad8c9ac66f823413a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39349 } } peers { permanent_uuid: "e728cd363cea4949a2a5d07380f215d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46533 } }
I20260926 20:49:17.171295 7827 ts_tablet_manager.cc:1428] T cc664b9806254d9abd018203d5a42ab7 P e728cd363cea4949a2a5d07380f215d8: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 20:49:17.171669 7465 heartbeater.cc:503] Master 127.0.98.60:41871 was elected leader, sending a full tablet report...
I20260926 20:49:17.171774 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:17.170367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40969 (local address 127.0.98.2:41523)
0926 20:49:17.171680 (+ 1313us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:17.171721 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1290,"thread_start_us":79,"threads_started":1}
I20260926 20:49:17.171944 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:17.171517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40969 (local address 127.0.98.1:35855)
0926 20:49:17.171868 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:17.171902 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":74,"threads_started":1}
I20260926 20:49:17.172008 7831 tablet_bootstrap.cc:492] T cc664b9806254d9abd018203d5a42ab7 P 345595a322944cad8c9ac66f823413a7: No bootstrap required, opened a new log
I20260926 20:49:17.172061 7831 ts_tablet_manager.cc:1397] T cc664b9806254d9abd018203d5a42ab7 P 345595a322944cad8c9ac66f823413a7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:49:17.172210 7831 raft_consensus.cc:348] T cc664b9806254d9abd018203d5a42ab7 P 345595a322944cad8c9ac66f823413a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01cba43c9838497f8d037dec2c16e44a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44443 } } peers { permanent_uuid: "345595a322944cad8c9ac66f823413a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39349 } } peers { permanent_uuid: "e728cd363cea4949a2a5d07380f215d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46533 } }
I20260926 20:49:17.172281 7831 raft_consensus.cc:374] T cc664b9806254d9abd018203d5a42ab7 P 345595a322944cad8c9ac66f823413a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:49:17.172302 7831 raft_consensus.cc:729] T cc664b9806254d9abd018203d5a42ab7 P 345595a322944cad8c9ac66f823413a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 345595a322944cad8c9ac66f823413a7, State: Initialized, Role: FOLLOWER
I20260926 20:49:17.172298 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:17.172007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40969 (local address 127.0.98.2:48941)
0926 20:49:17.172244 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:17.172273 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220}
I20260926 20:49:17.172308 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:17.172133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40969 (local address 127.0.98.1:60253)
0926 20:49:17.172241 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:17.172275 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":74,"threads_started":1}
I20260926 20:49:17.172353 7831 consensus_queue.cc:260] T cc664b9806254d9abd018203d5a42ab7 P 345595a322944cad8c9ac66f823413a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01cba43c9838497f8d037dec2c16e44a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44443 } } peers { permanent_uuid: "345595a322944cad8c9ac66f823413a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39349 } } peers { permanent_uuid: "e728cd363cea4949a2a5d07380f215d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46533 } }
I20260926 20:49:17.172672 7831 ts_tablet_manager.cc:1428] T cc664b9806254d9abd018203d5a42ab7 P 345595a322944cad8c9ac66f823413a7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:49:17.173043 7738 heartbeater.cc:503] Master 127.0.98.60:41871 was elected leader, sending a full tablet report...
I20260926 20:49:17.173096 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:17.172825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40969 (local address 127.0.98.3:47141)
0926 20:49:17.172941 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:17.172974 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":74,"threads_started":1}
I20260926 20:49:17.173426 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:17.173305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40969 (local address 127.0.98.3:49867)
0926 20:49:17.173378 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:17.173404 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260926 20:49:17.218168 7468 tablet.cc:2369] T cc664b9806254d9abd018203d5a42ab7 P e728cd363cea4949a2a5d07380f215d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:49:17.263955 7741 tablet.cc:2369] T cc664b9806254d9abd018203d5a42ab7 P 345595a322944cad8c9ac66f823413a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:49:17.337987 7837 raft_consensus.cc:482] T cc664b9806254d9abd018203d5a42ab7 P 345595a322944cad8c9ac66f823413a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:49:17.338089 7837 raft_consensus.cc:504] T cc664b9806254d9abd018203d5a42ab7 P 345595a322944cad8c9ac66f823413a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01cba43c9838497f8d037dec2c16e44a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44443 } } peers { permanent_uuid: "345595a322944cad8c9ac66f823413a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39349 } } peers { permanent_uuid: "e728cd363cea4949a2a5d07380f215d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46533 } }
I20260926 20:49:17.338399 7837 leader_election.cc:288] T cc664b9806254d9abd018203d5a42ab7 P 345595a322944cad8c9ac66f823413a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01cba43c9838497f8d037dec2c16e44a (127.0.98.2:44443), e728cd363cea4949a2a5d07380f215d8 (127.0.98.1:46533)
I20260926 20:49:17.341617 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:17.338608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44443 (local address 127.0.98.3:37685)
0926 20:49:17.338721 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:17.338733 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:17.338798 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:17.338979 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:17.338979 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:17.338982 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:49:17.339042 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:17.339044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:17.339727 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:17.339729 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:17.340558 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:17.340560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:17.341336 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:17.341336 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:17.341424 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:17.341427 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:49:17.341506 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 20:49:17.341545 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":20}
I20260926 20:49:17.341615 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:17.338652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37685 (local address 127.0.98.2:44443)
0926 20:49:17.338701 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:17.338703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:17.338789 (+ 86us) server_negotiation.cc:407] Connection header received
0926 20:49:17.338867 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:17.338868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:17.338880 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:17.338966 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:49:17.339132 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:17.339602 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:17.340647 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:17.341225 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:17.341289 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:17.341303 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:17.341519 (+ 216us) server_negotiation.cc:299] Negotiation successful
0926 20:49:17.341551 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 20:49:17.341724 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:17.338684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44403 (local address 127.0.98.1:46533)
0926 20:49:17.338747 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:17.338750 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:17.338931 (+ 181us) server_negotiation.cc:407] Connection header received
0926 20:49:17.338998 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:17.339000 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:17.339014 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:17.339099 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:49:17.339249 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:17.339836 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:17.340772 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:17.341355 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:17.341435 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:17.341446 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:17.341632 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 20:49:17.341666 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 20:49:17.341732 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:17.338617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46533 (local address 127.0.98.3:44403)
0926 20:49:17.338849 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:17.338859 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:17.338943 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:17.339104 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:17.339105 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:17.339107 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:49:17.339165 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:17.339167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:17.339912 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:17.339913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:17.340651 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:17.340653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:17.341475 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:17.341476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:17.341549 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:17.341552 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:49:17.341613 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 20:49:17.341652 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":49,"thread_start_us":132,"threads_started":1}
I20260926 20:49:17.342082 7556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc664b9806254d9abd018203d5a42ab7" candidate_uuid: "345595a322944cad8c9ac66f823413a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01cba43c9838497f8d037dec2c16e44a" is_pre_election: true
I20260926 20:49:17.342226 7556 raft_consensus.cc:2454] T cc664b9806254d9abd018203d5a42ab7 P 01cba43c9838497f8d037dec2c16e44a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 345595a322944cad8c9ac66f823413a7 in term 0.
I20260926 20:49:17.342306 7419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc664b9806254d9abd018203d5a42ab7" candidate_uuid: "345595a322944cad8c9ac66f823413a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e728cd363cea4949a2a5d07380f215d8" is_pre_election: true
I20260926 20:49:17.342422 7419 raft_consensus.cc:2454] T cc664b9806254d9abd018203d5a42ab7 P e728cd363cea4949a2a5d07380f215d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 345595a322944cad8c9ac66f823413a7 in term 0.
I20260926 20:49:17.342392 7628 leader_election.cc:302] T cc664b9806254d9abd018203d5a42ab7 P 345595a322944cad8c9ac66f823413a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01cba43c9838497f8d037dec2c16e44a, 345595a322944cad8c9ac66f823413a7; no voters:
I20260926 20:49:17.342514 7837 raft_consensus.cc:2781] T cc664b9806254d9abd018203d5a42ab7 P 345595a322944cad8c9ac66f823413a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:49:17.342550 7837 raft_consensus.cc:482] T cc664b9806254d9abd018203d5a42ab7 P 345595a322944cad8c9ac66f823413a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:49:17.342573 7837 raft_consensus.cc:3037] T cc664b9806254d9abd018203d5a42ab7 P 345595a322944cad8c9ac66f823413a7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:49:17.343245 7837 raft_consensus.cc:504] T cc664b9806254d9abd018203d5a42ab7 P 345595a322944cad8c9ac66f823413a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01cba43c9838497f8d037dec2c16e44a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44443 } } peers { permanent_uuid: "345595a322944cad8c9ac66f823413a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39349 } } peers { permanent_uuid: "e728cd363cea4949a2a5d07380f215d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46533 } }
I20260926 20:49:17.343400 7837 leader_election.cc:288] T cc664b9806254d9abd018203d5a42ab7 P 345595a322944cad8c9ac66f823413a7 [CANDIDATE]: Term 1 election: Requested vote from peers 01cba43c9838497f8d037dec2c16e44a (127.0.98.2:44443), e728cd363cea4949a2a5d07380f215d8 (127.0.98.1:46533)
I20260926 20:49:17.343538 7556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc664b9806254d9abd018203d5a42ab7" candidate_uuid: "345595a322944cad8c9ac66f823413a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01cba43c9838497f8d037dec2c16e44a"
I20260926 20:49:17.343549 7419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc664b9806254d9abd018203d5a42ab7" candidate_uuid: "345595a322944cad8c9ac66f823413a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e728cd363cea4949a2a5d07380f215d8"
I20260926 20:49:17.343624 7556 raft_consensus.cc:3037] T cc664b9806254d9abd018203d5a42ab7 P 01cba43c9838497f8d037dec2c16e44a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:49:17.343647 7419 raft_consensus.cc:3037] T cc664b9806254d9abd018203d5a42ab7 P e728cd363cea4949a2a5d07380f215d8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:49:17.344326 7556 raft_consensus.cc:2454] T cc664b9806254d9abd018203d5a42ab7 P 01cba43c9838497f8d037dec2c16e44a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 345595a322944cad8c9ac66f823413a7 in term 1.
I20260926 20:49:17.344326 7419 raft_consensus.cc:2454] T cc664b9806254d9abd018203d5a42ab7 P e728cd363cea4949a2a5d07380f215d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 345595a322944cad8c9ac66f823413a7 in term 1.
I20260926 20:49:17.344491 7628 leader_election.cc:302] T cc664b9806254d9abd018203d5a42ab7 P 345595a322944cad8c9ac66f823413a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01cba43c9838497f8d037dec2c16e44a, 345595a322944cad8c9ac66f823413a7; no voters:
I20260926 20:49:17.344595 7837 raft_consensus.cc:2781] T cc664b9806254d9abd018203d5a42ab7 P 345595a322944cad8c9ac66f823413a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:49:17.344755 7837 raft_consensus.cc:686] T cc664b9806254d9abd018203d5a42ab7 P 345595a322944cad8c9ac66f823413a7 [term 1 LEADER]: Becoming Leader. State: Replica: 345595a322944cad8c9ac66f823413a7, State: Running, Role: LEADER
I20260926 20:49:17.344837 7837 consensus_queue.cc:237] T cc664b9806254d9abd018203d5a42ab7 P 345595a322944cad8c9ac66f823413a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01cba43c9838497f8d037dec2c16e44a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44443 } } peers { permanent_uuid: "345595a322944cad8c9ac66f823413a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39349 } } peers { permanent_uuid: "e728cd363cea4949a2a5d07380f215d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46533 } }
I20260926 20:49:17.345083 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:17.344968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40969 (local address 127.0.98.3:45843)
0926 20:49:17.345005 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:17.345057 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260926 20:49:17.345604 7285 catalog_manager.cc:5495] T cc664b9806254d9abd018203d5a42ab7 P 345595a322944cad8c9ac66f823413a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 345595a322944cad8c9ac66f823413a7 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "345595a322944cad8c9ac66f823413a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01cba43c9838497f8d037dec2c16e44a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "345595a322944cad8c9ac66f823413a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e728cd363cea4949a2a5d07380f215d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46533 } health_report { overall_health: UNKNOWN } } }
W20260926 20:49:17.374003 7605 tablet.cc:2369] T cc664b9806254d9abd018203d5a42ab7 P 01cba43c9838497f8d037dec2c16e44a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:49:17.390334 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:17.388191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53560 (local address 127.0.98.3:39349)
0926 20:49:17.388247 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:17.388249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:17.388256 (+ 7us) server_negotiation.cc:407] Connection header received
0926 20:49:17.388290 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:17.388291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:17.388311 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:17.388372 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:49:17.388511 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:17.388986 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:17.389672 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:17.389847 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:17.389895 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:17.390027 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:49:17.390120 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:49:17.390171 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:17.390212 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 20:49:17.390229 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 20:49:17.391489 7419 raft_consensus.cc:1261] T cc664b9806254d9abd018203d5a42ab7 P e728cd363cea4949a2a5d07380f215d8 [term 1 FOLLOWER]: Refusing update from remote peer 345595a322944cad8c9ac66f823413a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:49:17.391501 7556 raft_consensus.cc:1261] T cc664b9806254d9abd018203d5a42ab7 P 01cba43c9838497f8d037dec2c16e44a [term 1 FOLLOWER]: Refusing update from remote peer 345595a322944cad8c9ac66f823413a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:49:17.391824 7837 consensus_queue.cc:1035] T cc664b9806254d9abd018203d5a42ab7 P 345595a322944cad8c9ac66f823413a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e728cd363cea4949a2a5d07380f215d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:49:17.391981 7840 consensus_queue.cc:1035] T cc664b9806254d9abd018203d5a42ab7 P 345595a322944cad8c9ac66f823413a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01cba43c9838497f8d037dec2c16e44a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:49:17.392727 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:17.392619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40969 (local address 127.0.98.1:35413)
0926 20:49:17.392662 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:17.392691 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260926 20:49:17.392992 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:17.392870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40969 (local address 127.0.98.2:46603)
0926 20:49:17.392913 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:17.392942 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260926 20:49:17.393756 7844 mvcc.cc:205] Tried to move back new op lower bound from 7333706782273040384 to 7333706782085234688. Current Snapshot: MvccSnapshot[applied={T|T < 7333706782273040384}]
I20260926 20:49:17.394112 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:17.393973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40969 (local address 127.0.98.2:56949)
0926 20:49:17.394056 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:17.394083 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 20:49:17.394604 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:17.394027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40969 (local address 127.0.98.3:40207)
0926 20:49:17.394533 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:17.394573 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490}
I20260926 20:49:17.394817 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:17.394724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40969 (local address 127.0.98.1:46245)
0926 20:49:17.394762 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:17.394786 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260926 20:49:17.394937 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:17.394836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40969 (local address 127.0.98.3:52903)
0926 20:49:17.394882 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:17.394910 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260926 20:49:17.397275 7285 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:49:17.397372 7285 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:49:17.398770 7285 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a19b05eba1c24e7db3c2ad66aa45fd33: Sending DeleteTablet for 3 replicas of tablet cc664b9806254d9abd018203d5a42ab7
I20260926 20:49:17.399053 7399 tablet_service.cc:1514] Processing DeleteTablet for tablet cc664b9806254d9abd018203d5a42ab7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:49:17 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:46620
I20260926 20:49:17.399102 7536 tablet_service.cc:1514] Processing DeleteTablet for tablet cc664b9806254d9abd018203d5a42ab7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:49:17 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:51002
I20260926 20:49:17.399154 7672 tablet_service.cc:1514] Processing DeleteTablet for tablet cc664b9806254d9abd018203d5a42ab7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:49:17 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:53552
I20260926 20:49:17.399328 7854 tablet_replica.cc:330] T cc664b9806254d9abd018203d5a42ab7 P e728cd363cea4949a2a5d07380f215d8: stopping tablet replica
I20260926 20:49:17.399392 7854 raft_consensus.cc:2229] T cc664b9806254d9abd018203d5a42ab7 P e728cd363cea4949a2a5d07380f215d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:49:17.399439 7854 raft_consensus.cc:2258] T cc664b9806254d9abd018203d5a42ab7 P e728cd363cea4949a2a5d07380f215d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:49:17.399555 7856 tablet_replica.cc:330] T cc664b9806254d9abd018203d5a42ab7 P 345595a322944cad8c9ac66f823413a7: stopping tablet replica
I20260926 20:49:17.399609 7856 raft_consensus.cc:2229] T cc664b9806254d9abd018203d5a42ab7 P 345595a322944cad8c9ac66f823413a7 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:49:17.399626 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 7241
I20260926 20:49:17.399690 7856 raft_consensus.cc:2258] T cc664b9806254d9abd018203d5a42ab7 P 345595a322944cad8c9ac66f823413a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:49:17.400230 7855 tablet_replica.cc:330] T cc664b9806254d9abd018203d5a42ab7 P 01cba43c9838497f8d037dec2c16e44a: stopping tablet replica
I20260926 20:49:17.400305 7855 raft_consensus.cc:2229] T cc664b9806254d9abd018203d5a42ab7 P 01cba43c9838497f8d037dec2c16e44a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:49:17.400354 7855 raft_consensus.cc:2258] T cc664b9806254d9abd018203d5a42ab7 P 01cba43c9838497f8d037dec2c16e44a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:49:17.401890 7855 ts_tablet_manager.cc:1905] T cc664b9806254d9abd018203d5a42ab7 P 01cba43c9838497f8d037dec2c16e44a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:49:17.403342 7856 ts_tablet_manager.cc:1905] T cc664b9806254d9abd018203d5a42ab7 P 345595a322944cad8c9ac66f823413a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:49:17.403873 7855 ts_tablet_manager.cc:1918] T cc664b9806254d9abd018203d5a42ab7 P 01cba43c9838497f8d037dec2c16e44a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:49:17.404023 7855 log.cc:1196] T cc664b9806254d9abd018203d5a42ab7 P 01cba43c9838497f8d037dec2c16e44a: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-1/wal/wals/cc664b9806254d9abd018203d5a42ab7
I20260926 20:49:17.404343 7855 ts_tablet_manager.cc:1939] T cc664b9806254d9abd018203d5a42ab7 P 01cba43c9838497f8d037dec2c16e44a: Deleting consensus metadata
I20260926 20:49:17.404708 7273 catalog_manager.cc:4841] TS 01cba43c9838497f8d037dec2c16e44a (127.0.98.2:44443): tablet cc664b9806254d9abd018203d5a42ab7 (table test-table [id=5255b402fb624242873bd0c452dcfeda]) successfully deleted
I20260926 20:49:17.404945 7856 ts_tablet_manager.cc:1918] T cc664b9806254d9abd018203d5a42ab7 P 345595a322944cad8c9ac66f823413a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:49:17.405014 7856 log.cc:1196] T cc664b9806254d9abd018203d5a42ab7 P 345595a322944cad8c9ac66f823413a7: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455725523895-392-0/minicluster-data/ts-2/wal/wals/cc664b9806254d9abd018203d5a42ab7
I20260926 20:49:17.405226 7856 ts_tablet_manager.cc:1939] T cc664b9806254d9abd018203d5a42ab7 P 345595a322944cad8c9ac66f823413a7: Deleting consensus metadata
I20260926 20:49:17.405596 7273 catalog_manager.cc:4841] TS 345595a322944cad8c9ac66f823413a7 (127.0.98.3:39349): tablet cc664b9806254d9abd018203d5a42ab7 (table test-table [id=5255b402fb624242873bd0c452dcfeda]) successfully deleted
W20260926 20:49:17.408133 7824 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:17.408009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46533 (local address 127.0.0.1:46622)
0926 20:49:17.408058 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:17.408083 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46533: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 20:49:17.408269 7271 catalog_manager.cc:4584] TS e728cd363cea4949a2a5d07380f215d8 (127.0.98.1:46533): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cc664b9806254d9abd018203d5a42ab7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46533: connect: Connection reset by peer (error 104)
I20260926 20:49:17.408360 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 7473
I20260926 20:49:17.415524 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 7609
I20260926 20:49:17.420130 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 6960
I20260926 20:49:17.424646 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 7148
2026-09-26T20:49:17Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4814 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:49:17 dist-test-slave-hwlj krb5kdc[7859](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:49:17 dist-test-slave-hwlj krb5kdc[7859](info): set up 2 sockets
Sep 26 20:49:17 dist-test-slave-hwlj krb5kdc[7859](info): commencing operation
krb5kdc: starting...
W20260926 20:49:18.464192 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/krb5kdc/test-user.keytab.
Sep 26 20:49:18 dist-test-slave-hwlj krb5kdc[7859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:49:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:49:18Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260926 20:49:18.583602 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32897
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:49:18.662967 7872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:18.663229 7872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:18.663267 7872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:18.664489 7872 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:49:18.664563 7872 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 20:49:18.664585 7872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:18.664608 7872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:18.664638 7872 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:49:18.664666 7872 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:49:18.666162 7872 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32897
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:18.666447 7872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:18.666673 7872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:18.671386 7878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:18.671464 7872 server_base.cc:1034] running on GCE node
W20260926 20:49:18.671396 7880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:18.671396 7877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:18.671905 7872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:18.672144 7872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:18.673318 7872 hybrid_clock.cc:648] HybridClock initialized: now 1790455758673307 us; error 34 us; skew 500 ppm
Sep 26 20:49:18 dist-test-slave-hwlj krb5kdc[7859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455758, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:49:18.675699 7872 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260926 20:49:18.676198 7872 webserver.cc:460] Webserver started at http://127.0.98.62:34705/ using document root <none> and password file <none>
I20260926 20:49:18.676417 7872 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:18.676486 7872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:18.676640 7872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:49:18.677686 7872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/master-0/data/instance:
uuid: "4b012d088b8243c290a80ca96476dd0c"
format_stamp: "Formatted at 2026-09-26 20:49:18 on dist-test-slave-hwlj"
server_key: "6e7d8ca19fce021aca9f3a7b94b51a49"
server_key_iv: "9e969266a131d5e8bcfc58154d864353"
server_key_version: "encryptionkey@0"
I20260926 20:49:18.678052 7872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/master-0/wal/instance:
uuid: "4b012d088b8243c290a80ca96476dd0c"
format_stamp: "Formatted at 2026-09-26 20:49:18 on dist-test-slave-hwlj"
server_key: "6e7d8ca19fce021aca9f3a7b94b51a49"
server_key_iv: "9e969266a131d5e8bcfc58154d864353"
server_key_version: "encryptionkey@0"
I20260926 20:49:18.679376 7872 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:49:18.680218 7887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:18.680435 7872 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 20:49:18.680534 7872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "4b012d088b8243c290a80ca96476dd0c"
format_stamp: "Formatted at 2026-09-26 20:49:18 on dist-test-slave-hwlj"
server_key: "6e7d8ca19fce021aca9f3a7b94b51a49"
server_key_iv: "9e969266a131d5e8bcfc58154d864353"
server_key_version: "encryptionkey@0"
I20260926 20:49:18.680612 7872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:18.691824 7872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:18.693929 7872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:18.694103 7872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:18.699004 7872 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32897
I20260926 20:49:18.699052 7949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32897 every 8 connection(s)
I20260926 20:49:18.699374 7872 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:49:18.700022 7950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:49:18.702240 7950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b012d088b8243c290a80ca96476dd0c: Bootstrap starting.
I20260926 20:49:18.702836 7950 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b012d088b8243c290a80ca96476dd0c: Neither blocks nor log segments found. Creating new log.
I20260926 20:49:18.703132 7950 log.cc:824] T 00000000000000000000000000000000 P 4b012d088b8243c290a80ca96476dd0c: Log is configured to *not* fsync() on all Append() calls
I20260926 20:49:18.703846 7950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b012d088b8243c290a80ca96476dd0c: No bootstrap required, opened a new log
I20260926 20:49:18.705899 7950 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b012d088b8243c290a80ca96476dd0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b012d088b8243c290a80ca96476dd0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32897 } }
I20260926 20:49:18.706037 7950 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b012d088b8243c290a80ca96476dd0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:49:18.706060 7950 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b012d088b8243c290a80ca96476dd0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b012d088b8243c290a80ca96476dd0c, State: Initialized, Role: FOLLOWER
I20260926 20:49:18.706209 7950 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b012d088b8243c290a80ca96476dd0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b012d088b8243c290a80ca96476dd0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32897 } }
I20260926 20:49:18.706284 7950 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b012d088b8243c290a80ca96476dd0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:49:18.706338 7950 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b012d088b8243c290a80ca96476dd0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:49:18.706385 7950 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b012d088b8243c290a80ca96476dd0c [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:49:18.706940 7950 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b012d088b8243c290a80ca96476dd0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b012d088b8243c290a80ca96476dd0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32897 } }
I20260926 20:49:18.707053 7950 leader_election.cc:302] T 00000000000000000000000000000000 P 4b012d088b8243c290a80ca96476dd0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b012d088b8243c290a80ca96476dd0c; no voters:
I20260926 20:49:18.707221 7950 leader_election.cc:288] T 00000000000000000000000000000000 P 4b012d088b8243c290a80ca96476dd0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:49:18.707247 7953 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b012d088b8243c290a80ca96476dd0c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:49:18.707414 7953 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b012d088b8243c290a80ca96476dd0c [term 1 LEADER]: Becoming Leader. State: Replica: 4b012d088b8243c290a80ca96476dd0c, State: Running, Role: LEADER
I20260926 20:49:18.707499 7950 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b012d088b8243c290a80ca96476dd0c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:49:18.707504 7953 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b012d088b8243c290a80ca96476dd0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b012d088b8243c290a80ca96476dd0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32897 } }
I20260926 20:49:18.708001 7955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b012d088b8243c290a80ca96476dd0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b012d088b8243c290a80ca96476dd0c. Latest consensus state: current_term: 1 leader_uuid: "4b012d088b8243c290a80ca96476dd0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b012d088b8243c290a80ca96476dd0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32897 } } }
I20260926 20:49:18.708007 7954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b012d088b8243c290a80ca96476dd0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b012d088b8243c290a80ca96476dd0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b012d088b8243c290a80ca96476dd0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32897 } } }
I20260926 20:49:18.708129 7955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b012d088b8243c290a80ca96476dd0c [sys.catalog]: This master's current role is: LEADER
I20260926 20:49:18.708134 7954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b012d088b8243c290a80ca96476dd0c [sys.catalog]: This master's current role is: LEADER
I20260926 20:49:18.708478 7962 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:49:18.708506 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 7872
I20260926 20:49:18.708659 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/master-0/wal/instance
I20260926 20:49:18.708889 392 external_mini_cluster.cc:1428] Setting key 4457a68bb5e42830e0b51051be9f3063
I20260926 20:49:18.709055 7962 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:49:18.710819 7962 catalog_manager.cc:1347] Generated new cluster ID: 4b26a02ea218467a9385d3d693b59dc7
I20260926 20:49:18.710873 7962 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 20:49:18 dist-test-slave-hwlj krb5kdc[7859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 20:49:18.718750 7972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:18.709498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49720 (local address 127.0.98.62:32897)
0926 20:49:18.709676 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:18.709685 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:18.709717 (+ 32us) server_negotiation.cc:407] Connection header received
0926 20:49:18.709840 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:18.709846 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:18.709908 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:18.710064 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:18.710804 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:18.711851 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:18.712659 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:18.713080 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:18.713209 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:18.715489 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:18.715493 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:18.715495 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:18.715512 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:18.717212 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:18.717601 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:18.717604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:18.717604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:18.717696 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:18.717903 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:18.717904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:18.717904 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:18.718154 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:18.718322 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:18.718419 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 20:49:18.718577 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":79,"threads_started":1}
I20260926 20:49:18.732507 7962 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:49:18.733111 7962 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:49:18.737668 7962 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b012d088b8243c290a80ca96476dd0c: Generated new TSK 0
I20260926 20:49:18.737869 7962 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260926 20:49:18.770612 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32897
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:49:18.859108 7976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:18.859321 7976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:18.859346 7976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:18.860564 7976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:18.860605 7976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:18.860626 7976 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:49:18.860687 7976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:49:18.862210 7976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:32897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:18.862478 7976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:18.862711 7976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:49:18.867939 7976 server_base.cc:1034] running on GCE node
W20260926 20:49:18.867895 7981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:18.867889 7984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:18.868083 7982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:18.868291 7976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:18.868530 7976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:18.869697 7976 hybrid_clock.cc:648] HybridClock initialized: now 1790455758869684 us; error 27 us; skew 500 ppm
Sep 26 20:49:18 dist-test-slave-hwlj krb5kdc[7859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455758, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:49:18.872313 7976 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260926 20:49:18.872782 7976 webserver.cc:460] Webserver started at http://127.0.98.1:34575/ using document root <none> and password file <none>
I20260926 20:49:18.873003 7976 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:18.873059 7976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:18.873149 7976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:49:18.874154 7976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-0/data/instance:
uuid: "b7762c6bd65f4238ac92779f523d6b1c"
format_stamp: "Formatted at 2026-09-26 20:49:18 on dist-test-slave-hwlj"
server_key: "8769e47d992219769c8b80e908a1370c"
server_key_iv: "d63c019f13d072697a4873b61c929d36"
server_key_version: "encryptionkey@0"
I20260926 20:49:18.874462 7976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b7762c6bd65f4238ac92779f523d6b1c"
format_stamp: "Formatted at 2026-09-26 20:49:18 on dist-test-slave-hwlj"
server_key: "8769e47d992219769c8b80e908a1370c"
server_key_iv: "d63c019f13d072697a4873b61c929d36"
server_key_version: "encryptionkey@0"
I20260926 20:49:18.875684 7976 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 20:49:18.876472 7991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:18.876626 7976 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:49:18.876730 7976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-0/wal
uuid: "b7762c6bd65f4238ac92779f523d6b1c"
format_stamp: "Formatted at 2026-09-26 20:49:18 on dist-test-slave-hwlj"
server_key: "8769e47d992219769c8b80e908a1370c"
server_key_iv: "d63c019f13d072697a4873b61c929d36"
server_key_version: "encryptionkey@0"
I20260926 20:49:18.876804 7976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:18.900707 7976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:18.903163 7976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:18.903332 7976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:18.903970 7976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:49:18.904027 7976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:18.904058 7976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:49:18.904080 7976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:18.911608 7976 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38323
I20260926 20:49:18.911703 8106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38323 every 8 connection(s)
Sep 26 20:49:18 dist-test-slave-hwlj krb5kdc[7859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455758, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 20:49:18.912074 7976 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:49:18.915354 7972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:18.904443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39375 (local address 127.0.98.62:32897)
0926 20:49:18.904515 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:18.904517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:18.904766 (+ 249us) server_negotiation.cc:407] Connection header received
0926 20:49:18.905331 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:18.905333 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:18.905352 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:18.905435 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:18.906195 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:18.907181 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:18.908680 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:18.908901 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:18.909201 (+ 300us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:18.912593 (+ 3392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:18.912594 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:18.912595 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:18.912603 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:18.914100 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:18.914496 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:18.914497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:18.914497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:18.914561 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:18.914829 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:18.914830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:18.914830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:18.914932 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:18.914980 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:18.915222 (+ 242us) server_negotiation.cc:299] Negotiation successful
0926 20:49:18.915290 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 20:49:18.915526 8000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:18.904418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32897 (local address 127.0.98.1:39375)
0926 20:49:18.904615 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:18.904661 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:18.905216 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:18.905526 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:18.905529 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:18.905853 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:18.906078 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:18.906089 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:18.907332 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:18.907334 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:18.908563 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:18.908566 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:18.909221 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:18.909222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:18.909417 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:18.910533 (+ 1116us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:18.910545 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:18.912495 (+ 1950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:18.914187 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:18.914192 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:18.914194 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:18.914428 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:18.914649 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:18.914650 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:18.914650 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:18.914768 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:18.914990 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:18.914994 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:18.915110 (+ 116us) client_negotiation.cc:769] Sending connection context
0926 20:49:18.915194 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 20:49:18.915319 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":112,"threads_started":1}
I20260926 20:49:18.916064 8107 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32897
I20260926 20:49:18.916152 8107 heartbeater.cc:464] Registering TS with master...
I20260926 20:49:18.916370 8107 heartbeater.cc:511] Master 127.0.98.62:32897 requested a full tablet report, sending...
I20260926 20:49:18.916481 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 7976
I20260926 20:49:18.916594 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-0/wal/instance
I20260926 20:49:18.916805 392 external_mini_cluster.cc:1428] Setting key ad43ce57b308335cb6a1aac3228b1d26
I20260926 20:49:18.917723 7904 ts_manager.cc:194] Registered new tserver with Master: b7762c6bd65f4238ac92779f523d6b1c (127.0.98.1:38323)
I20260926 20:49:18.918644 7904 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39375
I20260926 20:49:18.923586 7972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:18.917114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60833 (local address 127.0.98.62:32897)
0926 20:49:18.917154 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:18.917156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:18.917277 (+ 121us) server_negotiation.cc:407] Connection header received
0926 20:49:18.917314 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:18.917314 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:18.917324 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:18.917379 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:18.917759 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:18.918567 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:18.919679 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:18.919922 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:18.919979 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:18.920956 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:18.920957 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:18.920958 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:18.920965 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:18.922341 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:18.922772 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:18.922773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:18.922774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:18.922829 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:18.923050 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:18.923051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:18.923051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:18.923168 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:18.923239 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:18.923420 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 20:49:18.923497 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":19}
I20260926 20:49:18.923588 8113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:18.917032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32897 (local address 127.0.98.1:60833)
0926 20:49:18.917211 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:18.917220 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:18.917266 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:18.917381 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:18.917382 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:18.917595 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:18.917681 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:18.917683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:18.918677 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:18.918678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:18.919592 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:18.919595 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:18.920003 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:18.920004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:18.920071 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:18.920584 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:18.920588 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:18.920864 (+ 276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:18.922499 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:18.922500 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:18.922501 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:18.922685 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:18.922903 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:18.922904 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:18.922904 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:18.922985 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:18.923245 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:18.923246 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:18.923288 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 20:49:18.923377 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 20:49:18.923488 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":71,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260926 20:49:18.945037 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32897
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:49:19.029276 8116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:19.029479 8116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:19.029502 8116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:19.030655 8116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:19.030697 8116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:19.030717 8116 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:49:19.030766 8116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:49:19.032240 8116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:32897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:19.032467 8116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:19.032711 8116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:19.038156 8122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:19.038246 8116 server_base.cc:1034] running on GCE node
W20260926 20:49:19.038165 8121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:19.038165 8124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:19.038673 8116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:19.039036 8116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:19.040230 8116 hybrid_clock.cc:648] HybridClock initialized: now 1790455759040194 us; error 54 us; skew 500 ppm
Sep 26 20:49:19 dist-test-slave-hwlj krb5kdc[7859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455759, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:49:19.046814 8116 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260926 20:49:19.047318 8116 webserver.cc:460] Webserver started at http://127.0.98.2:39389/ using document root <none> and password file <none>
I20260926 20:49:19.047550 8116 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:19.047619 8116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:19.047819 8116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:49:19.048914 8116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-1/data/instance:
uuid: "5ebea5fcb4064638863a64bdbf8bb189"
format_stamp: "Formatted at 2026-09-26 20:49:19 on dist-test-slave-hwlj"
server_key: "349006ef69d7eb343c64844d65eebc3f"
server_key_iv: "81fd6a038e3f76fa6c9deb7f1abab2c5"
server_key_version: "encryptionkey@0"
I20260926 20:49:19.049286 8116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5ebea5fcb4064638863a64bdbf8bb189"
format_stamp: "Formatted at 2026-09-26 20:49:19 on dist-test-slave-hwlj"
server_key: "349006ef69d7eb343c64844d65eebc3f"
server_key_iv: "81fd6a038e3f76fa6c9deb7f1abab2c5"
server_key_version: "encryptionkey@0"
I20260926 20:49:19.050648 8116 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:49:19.051441 8131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:19.051656 8116 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:49:19.051787 8116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-1/wal
uuid: "5ebea5fcb4064638863a64bdbf8bb189"
format_stamp: "Formatted at 2026-09-26 20:49:19 on dist-test-slave-hwlj"
server_key: "349006ef69d7eb343c64844d65eebc3f"
server_key_iv: "81fd6a038e3f76fa6c9deb7f1abab2c5"
server_key_version: "encryptionkey@0"
I20260926 20:49:19.051882 8116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:19.067628 8116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:19.070152 8116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:19.070307 8116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:19.070815 8116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:49:19.070883 8116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:19.070930 8116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:49:19.070958 8116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:19.077752 8116 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43131
Sep 26 20:49:19 dist-test-slave-hwlj krb5kdc[7859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455759, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 20:49:19.077865 8246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43131 every 8 connection(s)
I20260926 20:49:19.078135 8116 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
I20260926 20:49:19.080094 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 8116
I20260926 20:49:19.080225 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-1/wal/instance
I20260926 20:49:19.080456 392 external_mini_cluster.cc:1428] Setting key 1eba2cc543fdc11e164eae674fc49615
I20260926 20:49:19.081660 8140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:19.071276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32897 (local address 127.0.98.2:55035)
0926 20:49:19.071440 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:19.071479 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:19.072051 (+ 572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:19.072359 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:19.072363 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:19.072699 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:19.072981 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:19.072988 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.074019 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.074020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:19.075375 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:19.075378 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.075900 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.075901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:19.076035 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:19.076590 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:19.076598 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:19.078269 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:19.079979 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:19.079987 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:19.079989 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:19.080311 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:19.080591 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:19.080592 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:19.080592 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:19.080706 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:19.081104 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:19.081110 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:19.081275 (+ 165us) client_negotiation.cc:769] Sending connection context
0926 20:49:19.081375 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 20:49:19.081497 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":85,"threads_started":1}
I20260926 20:49:19.081660 7972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:19.071305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55035 (local address 127.0.98.62:32897)
0926 20:49:19.071380 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:19.071383 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:19.071571 (+ 188us) server_negotiation.cc:407] Connection header received
0926 20:49:19.072151 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:19.072153 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:19.072168 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:19.072247 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:19.073088 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.073896 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.075460 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.075823 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.075893 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:19.078425 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:19.078427 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:19.078427 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:19.078435 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:19.079903 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:19.080412 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:19.080414 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:19.080414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:19.080498 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:19.080850 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:19.080851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:19.080851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:19.081016 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:19.081099 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:19.081427 (+ 328us) server_negotiation.cc:299] Negotiation successful
0926 20:49:19.081523 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 20:49:19.082208 8247 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32897
I20260926 20:49:19.082296 8247 heartbeater.cc:464] Registering TS with master...
I20260926 20:49:19.082491 8247 heartbeater.cc:511] Master 127.0.98.62:32897 requested a full tablet report, sending...
I20260926 20:49:19.082859 7903 ts_manager.cc:194] Registered new tserver with Master: 5ebea5fcb4064638863a64bdbf8bb189 (127.0.98.2:43131)
I20260926 20:49:19.083385 7903 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55035
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 20:49:19.090003 7972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:19.083363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38887 (local address 127.0.98.62:32897)
0926 20:49:19.083428 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:19.083430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:19.084104 (+ 674us) server_negotiation.cc:407] Connection header received
0926 20:49:19.084139 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:19.084140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:19.084149 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:19.084191 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:19.084588 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.085046 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.086150 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.086323 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.086373 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:19.087457 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:19.087458 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:19.087459 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:19.087464 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:19.088863 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:19.089251 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:19.089251 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:19.089252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:19.089299 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:19.089500 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:19.089500 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:19.089501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:19.089622 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:19.089671 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:19.089827 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 20:49:19.089911 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 20:49:19.090044 8254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:19.083254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32897 (local address 127.0.98.2:38887)
0926 20:49:19.084001 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:19.084013 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:19.084078 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:19.084196 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:19.084197 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:19.084424 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:19.084486 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:19.084488 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.085149 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.085151 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:19.086072 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:19.086075 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.086408 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.086409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:19.086495 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:19.086909 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:19.086915 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:19.087357 (+ 442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:19.088945 (+ 1588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:19.088946 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:19.088947 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:19.089173 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:19.089362 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:19.089363 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:19.089363 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:19.089439 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:19.089685 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:19.089686 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:19.089730 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 20:49:19.089803 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 20:49:19.089919 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"thread_start_us":68,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260926 20:49:19.112587 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32897
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:49:19.197079 8256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:19.197291 8256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:19.197317 8256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:19.198418 8256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:19.198482 8256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:19.198513 8256 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:49:19.198537 8256 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:49:19.200511 8256 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:32897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8256
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:19.200703 8256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:19.200877 8256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:19.205873 8261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:19.205929 8256 server_base.cc:1034] running on GCE node
W20260926 20:49:19.205896 8262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:19.205945 8264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:19.206228 8256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:19.206525 8256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:19.207702 8256 hybrid_clock.cc:648] HybridClock initialized: now 1790455759207660 us; error 56 us; skew 500 ppm
Sep 26 20:49:19 dist-test-slave-hwlj krb5kdc[7859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455759, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:49:19.210588 8256 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260926 20:49:19.211126 8256 webserver.cc:460] Webserver started at http://127.0.98.3:39021/ using document root <none> and password file <none>
I20260926 20:49:19.211377 8256 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:19.211455 8256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:19.211613 8256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:49:19.212776 8256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-2/data/instance:
uuid: "aad2e8806cab4a71bcdc7e7372c36854"
format_stamp: "Formatted at 2026-09-26 20:49:19 on dist-test-slave-hwlj"
server_key: "423b3318e76db8dbcf04adda346fa625"
server_key_iv: "2118a77d030a2db33a13e17ee8090758"
server_key_version: "encryptionkey@0"
I20260926 20:49:19.213146 8256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-2/wal/instance:
uuid: "aad2e8806cab4a71bcdc7e7372c36854"
format_stamp: "Formatted at 2026-09-26 20:49:19 on dist-test-slave-hwlj"
server_key: "423b3318e76db8dbcf04adda346fa625"
server_key_iv: "2118a77d030a2db33a13e17ee8090758"
server_key_version: "encryptionkey@0"
I20260926 20:49:19.214538 8256 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:49:19.215307 8271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:19.215500 8256 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:49:19.215615 8256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-2/wal
uuid: "aad2e8806cab4a71bcdc7e7372c36854"
format_stamp: "Formatted at 2026-09-26 20:49:19 on dist-test-slave-hwlj"
server_key: "423b3318e76db8dbcf04adda346fa625"
server_key_iv: "2118a77d030a2db33a13e17ee8090758"
server_key_version: "encryptionkey@0"
I20260926 20:49:19.215690 8256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:19.236889 8256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:19.239666 8256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:19.239907 8256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:19.240614 8256 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:49:19.240688 8256 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:19.240734 8256 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:49:19.240775 8256 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 20:49:19 dist-test-slave-hwlj krb5kdc[7859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455759, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 20:49:19.247996 8256 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37051
I20260926 20:49:19.248042 8386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37051 every 8 connection(s)
I20260926 20:49:19.248435 8256 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
I20260926 20:49:19.250589 7972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:19.240976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51085 (local address 127.0.98.62:32897)
0926 20:49:19.241031 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:19.241034 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:19.241258 (+ 224us) server_negotiation.cc:407] Connection header received
0926 20:49:19.241814 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:19.241815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:19.241828 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:19.241875 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:19.242736 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.243577 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.244683 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.245037 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.245122 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:19.247415 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:19.247417 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:19.247417 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:19.247425 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:19.249176 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:19.249618 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:19.249620 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:19.249620 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:19.249685 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:19.249932 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:19.249933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:19.249934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:19.250074 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:19.250149 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:19.250416 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 20:49:19.250493 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 20:49:19.250785 8280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:19.241002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32897 (local address 127.0.98.3:51085)
0926 20:49:19.241146 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:19.241185 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:19.241708 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:19.241960 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:19.241963 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:19.242307 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:19.242612 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:19.242619 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.243685 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.243686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:19.244573 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:19.244576 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.245145 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.245147 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:19.245275 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:19.245860 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:19.245868 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:19.247307 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:19.249278 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:19.249285 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:19.249287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:19.249526 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:19.249752 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:19.249753 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:19.249753 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:19.249859 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:19.250141 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:19.250147 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:19.250286 (+ 139us) client_negotiation.cc:769] Sending connection context
0926 20:49:19.250394 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 20:49:19.250563 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":80,"threads_started":1}
I20260926 20:49:19.251255 8387 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32897
I20260926 20:49:19.251341 8387 heartbeater.cc:464] Registering TS with master...
I20260926 20:49:19.251526 8387 heartbeater.cc:511] Master 127.0.98.62:32897 requested a full tablet report, sending...
I20260926 20:49:19.251940 7904 ts_manager.cc:194] Registered new tserver with Master: aad2e8806cab4a71bcdc7e7372c36854 (127.0.98.3:37051)
I20260926 20:49:19.252502 7904 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51085
I20260926 20:49:19.257679 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 8256
I20260926 20:49:19.257787 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-2/wal/instance
I20260926 20:49:19.257961 392 external_mini_cluster.cc:1428] Setting key 68111932cd4792f1e52e87f01e458c0f
I20260926 20:49:19.258061 7972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:19.252478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44839 (local address 127.0.98.62:32897)
0926 20:49:19.252518 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:19.252520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:19.252580 (+ 60us) server_negotiation.cc:407] Connection header received
0926 20:49:19.252674 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:19.252675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:19.252685 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:19.252742 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:19.253083 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.253556 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.254316 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.254521 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.254621 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:19.255659 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:19.255661 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:19.255661 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:19.255667 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:19.256893 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:19.257285 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:19.257287 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:19.257287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:19.257348 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:19.257546 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:19.257547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:19.257547 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:19.257659 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:19.257733 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:19.257897 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 20:49:19.257967 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 20:49:19.258076 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:19.252389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32897 (local address 127.0.98.3:44839)
0926 20:49:19.252518 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:19.252529 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:19.252592 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:19.252740 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:19.252740 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:19.252931 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:19.252989 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:19.252990 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.253648 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.253650 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:19.254209 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:19.254211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.254623 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.254624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:19.254693 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:19.255250 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:19.255255 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:19.255534 (+ 279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:19.257010 (+ 1476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:19.257011 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:19.257011 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:19.257194 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:19.257418 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:19.257419 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:19.257419 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:19.257486 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:19.257737 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:19.257738 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:19.257783 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 20:49:19.257849 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 20:49:19.257969 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":78,"threads_started":1}
I20260926 20:49:19.258880 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 20:49:19 dist-test-slave-hwlj krb5kdc[7859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455759, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 20:49:19 dist-test-slave-hwlj krb5kdc[7859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:49:19 dist-test-slave-hwlj krb5kdc[7859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 20:49:19.281805 7972 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:19.274268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49724 (local address 127.0.98.62:32897)
0926 20:49:19.274314 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:19.274316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:19.274464 (+ 148us) server_negotiation.cc:407] Connection header received
0926 20:49:19.274506 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:19.274507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:19.274517 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:19.274593 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:19.275425 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.276216 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.277508 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.277850 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.277895 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:19.281725 (+ 3830us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49724: BlockingRecv error: recv got EOF from 127.0.0.1:49724 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 26 20:49:19 dist-test-slave-hwlj krb5kdc[7859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455759, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 20:49:19.291555 7972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:19.283719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45862 (local address 127.0.98.62:32897)
0926 20:49:19.283781 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:19.283783 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:19.283881 (+ 98us) server_negotiation.cc:407] Connection header received
0926 20:49:19.283933 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:19.283934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:19.283945 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:19.283998 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:19.284749 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.285352 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.286298 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.286574 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.286646 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:19.289346 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:19.289347 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:19.289347 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:19.289363 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:19.290667 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:19.290989 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:19.290990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:19.290990 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:19.291039 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:19.291186 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:19.291186 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:19.291187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:19.291278 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:19.291346 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:19.291428 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 20:49:19.291488 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 20:49:19.293169 7903 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45862:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:49:19.294739 7903 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:49:19.301389 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:19.297250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37051 (local address 127.0.0.1:33166)
0926 20:49:19.297516 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:19.297528 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:19.297579 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:19.298005 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:19.298006 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:19.298009 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:49:19.298055 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:19.298056 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.299275 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.299276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:19.300375 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:19.300377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.301084 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.301085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:19.301156 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:19.301170 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 20:49:19.301248 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 20:49:19.301300 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":91,"threads_started":1}
I20260926 20:49:19.301412 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:19.297338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33166 (local address 127.0.98.3:37051)
0926 20:49:19.297808 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:19.297813 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:19.297822 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:49:19.297863 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:19.297865 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:19.297880 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:19.298123 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:49:19.298179 (+ 56us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.299175 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.300463 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.300995 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.301086 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:19.301180 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:19.301264 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 20:49:19.301302 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":74,"threads_started":1}
I20260926 20:49:19.301510 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:19.297228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43131 (local address 127.0.0.1:48720)
0926 20:49:19.297417 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:19.297463 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:19.297528 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:19.297832 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:19.297835 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:19.297837 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:49:19.297908 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:19.297910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.298934 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.298936 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:19.300478 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:19.300480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.301309 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.301309 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:19.301376 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:19.301378 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 20:49:19.301404 (+ 26us) client_negotiation.cc:240] Negotiation successful
0926 20:49:19.301443 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":63,"threads_started":1}
I20260926 20:49:19.301652 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:19.297430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38323 (local address 127.0.0.1:53438)
0926 20:49:19.297630 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:19.297639 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:19.297676 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:19.297974 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:19.297975 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:19.297977 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:49:19.298023 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:19.298024 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.299489 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.299490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:19.300329 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:19.300331 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.301250 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.301251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:19.301321 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:19.301325 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:49:19.301354 (+ 29us) client_negotiation.cc:240] Negotiation successful
0926 20:49:19.301399 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":64,"threads_started":1}
I20260926 20:49:19.301715 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:19.297645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53438 (local address 127.0.98.1:38323)
0926 20:49:19.297808 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:19.297813 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:19.297823 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:49:19.297862 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:19.297864 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:19.297880 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:19.297965 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:49:19.298412 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.299395 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.300429 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.301145 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.301446 (+ 301us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:19.301538 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:19.301565 (+ 27us) server_negotiation.cc:299] Negotiation successful
0926 20:49:19.301611 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":81,"threads_started":1}
I20260926 20:49:19.301927 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:19.297321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48720 (local address 127.0.98.2:43131)
0926 20:49:19.297671 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:19.297677 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:19.297688 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:49:19.297730 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:19.297733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:19.297749 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:19.297830 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:49:19.298018 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.298851 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.300549 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.301228 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.301678 (+ 450us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:19.301768 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:19.301794 (+ 26us) server_negotiation.cc:299] Negotiation successful
0926 20:49:19.301840 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":76,"threads_started":1}
I20260926 20:49:19.302460 8041 tablet_service.cc:1467] Processing CreateTablet for tablet 2e5de8763ff94675953913c917b16d68 (DEFAULT_TABLE table=test-table [id=2ddfe76eeec54521bebe4315052f16ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:49:19.302528 8181 tablet_service.cc:1467] Processing CreateTablet for tablet 2e5de8763ff94675953913c917b16d68 (DEFAULT_TABLE table=test-table [id=2ddfe76eeec54521bebe4315052f16ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:49:19.302840 8181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e5de8763ff94675953913c917b16d68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:49:19.302845 8041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e5de8763ff94675953913c917b16d68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:49:19.304387 8321 tablet_service.cc:1467] Processing CreateTablet for tablet 2e5de8763ff94675953913c917b16d68 (DEFAULT_TABLE table=test-table [id=2ddfe76eeec54521bebe4315052f16ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:49:19.304711 8321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e5de8763ff94675953913c917b16d68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:49:19.306195 8415 tablet_bootstrap.cc:492] T 2e5de8763ff94675953913c917b16d68 P 5ebea5fcb4064638863a64bdbf8bb189: Bootstrap starting.
I20260926 20:49:19.306789 8415 tablet_bootstrap.cc:654] T 2e5de8763ff94675953913c917b16d68 P 5ebea5fcb4064638863a64bdbf8bb189: Neither blocks nor log segments found. Creating new log.
I20260926 20:49:19.307108 8415 log.cc:824] T 2e5de8763ff94675953913c917b16d68 P 5ebea5fcb4064638863a64bdbf8bb189: Log is configured to *not* fsync() on all Append() calls
I20260926 20:49:19.307857 8415 tablet_bootstrap.cc:492] T 2e5de8763ff94675953913c917b16d68 P 5ebea5fcb4064638863a64bdbf8bb189: No bootstrap required, opened a new log
I20260926 20:49:19.307930 8415 ts_tablet_manager.cc:1397] T 2e5de8763ff94675953913c917b16d68 P 5ebea5fcb4064638863a64bdbf8bb189: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:49:19.307930 8416 tablet_bootstrap.cc:492] T 2e5de8763ff94675953913c917b16d68 P b7762c6bd65f4238ac92779f523d6b1c: Bootstrap starting.
I20260926 20:49:19.308276 8418 tablet_bootstrap.cc:492] T 2e5de8763ff94675953913c917b16d68 P aad2e8806cab4a71bcdc7e7372c36854: Bootstrap starting.
I20260926 20:49:19.308612 8416 tablet_bootstrap.cc:654] T 2e5de8763ff94675953913c917b16d68 P b7762c6bd65f4238ac92779f523d6b1c: Neither blocks nor log segments found. Creating new log.
I20260926 20:49:19.308924 8416 log.cc:824] T 2e5de8763ff94675953913c917b16d68 P b7762c6bd65f4238ac92779f523d6b1c: Log is configured to *not* fsync() on all Append() calls
I20260926 20:49:19.309055 8418 tablet_bootstrap.cc:654] T 2e5de8763ff94675953913c917b16d68 P aad2e8806cab4a71bcdc7e7372c36854: Neither blocks nor log segments found. Creating new log.
I20260926 20:49:19.309391 8418 log.cc:824] T 2e5de8763ff94675953913c917b16d68 P aad2e8806cab4a71bcdc7e7372c36854: Log is configured to *not* fsync() on all Append() calls
I20260926 20:49:19.309901 8415 raft_consensus.cc:348] T 2e5de8763ff94675953913c917b16d68 P 5ebea5fcb4064638863a64bdbf8bb189 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } } peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } }
I20260926 20:49:19.310031 8415 raft_consensus.cc:374] T 2e5de8763ff94675953913c917b16d68 P 5ebea5fcb4064638863a64bdbf8bb189 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:49:19.310055 8415 raft_consensus.cc:729] T 2e5de8763ff94675953913c917b16d68 P 5ebea5fcb4064638863a64bdbf8bb189 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ebea5fcb4064638863a64bdbf8bb189, State: Initialized, Role: FOLLOWER
I20260926 20:49:19.310091 8418 tablet_bootstrap.cc:492] T 2e5de8763ff94675953913c917b16d68 P aad2e8806cab4a71bcdc7e7372c36854: No bootstrap required, opened a new log
I20260926 20:49:19.310154 8418 ts_tablet_manager.cc:1397] T 2e5de8763ff94675953913c917b16d68 P aad2e8806cab4a71bcdc7e7372c36854: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:49:19.310187 8415 consensus_queue.cc:260] T 2e5de8763ff94675953913c917b16d68 P 5ebea5fcb4064638863a64bdbf8bb189 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } } peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } }
I20260926 20:49:19.310483 8415 ts_tablet_manager.cc:1428] T 2e5de8763ff94675953913c917b16d68 P 5ebea5fcb4064638863a64bdbf8bb189: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 20:49:19.310520 8247 heartbeater.cc:503] Master 127.0.98.62:32897 was elected leader, sending a full tablet report...
I20260926 20:49:19.311071 8416 tablet_bootstrap.cc:492] T 2e5de8763ff94675953913c917b16d68 P b7762c6bd65f4238ac92779f523d6b1c: No bootstrap required, opened a new log
I20260926 20:49:19.311120 8416 ts_tablet_manager.cc:1397] T 2e5de8763ff94675953913c917b16d68 P b7762c6bd65f4238ac92779f523d6b1c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 20:49:19.312199 8418 raft_consensus.cc:348] T 2e5de8763ff94675953913c917b16d68 P aad2e8806cab4a71bcdc7e7372c36854 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } } peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } }
I20260926 20:49:19.312354 8418 raft_consensus.cc:374] T 2e5de8763ff94675953913c917b16d68 P aad2e8806cab4a71bcdc7e7372c36854 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:49:19.312402 8418 raft_consensus.cc:729] T 2e5de8763ff94675953913c917b16d68 P aad2e8806cab4a71bcdc7e7372c36854 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad2e8806cab4a71bcdc7e7372c36854, State: Initialized, Role: FOLLOWER
I20260926 20:49:19.312515 8418 consensus_queue.cc:260] T 2e5de8763ff94675953913c917b16d68 P aad2e8806cab4a71bcdc7e7372c36854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } } peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } }
I20260926 20:49:19.312764 8418 ts_tablet_manager.cc:1428] T 2e5de8763ff94675953913c917b16d68 P aad2e8806cab4a71bcdc7e7372c36854: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:49:19.312789 8387 heartbeater.cc:503] Master 127.0.98.62:32897 was elected leader, sending a full tablet report...
I20260926 20:49:19.312865 8416 raft_consensus.cc:348] T 2e5de8763ff94675953913c917b16d68 P b7762c6bd65f4238ac92779f523d6b1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } } peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } }
I20260926 20:49:19.313025 8416 raft_consensus.cc:374] T 2e5de8763ff94675953913c917b16d68 P b7762c6bd65f4238ac92779f523d6b1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:49:19.313076 8416 raft_consensus.cc:729] T 2e5de8763ff94675953913c917b16d68 P b7762c6bd65f4238ac92779f523d6b1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7762c6bd65f4238ac92779f523d6b1c, State: Initialized, Role: FOLLOWER
I20260926 20:49:19.313187 8416 consensus_queue.cc:260] T 2e5de8763ff94675953913c917b16d68 P b7762c6bd65f4238ac92779f523d6b1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } } peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } }
I20260926 20:49:19.313452 8416 ts_tablet_manager.cc:1428] T 2e5de8763ff94675953913c917b16d68 P b7762c6bd65f4238ac92779f523d6b1c: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260926 20:49:19.313470 8107 heartbeater.cc:503] Master 127.0.98.62:32897 was elected leader, sending a full tablet report...
I20260926 20:49:19.328011 8422 raft_consensus.cc:482] T 2e5de8763ff94675953913c917b16d68 P aad2e8806cab4a71bcdc7e7372c36854 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:49:19.328112 8422 raft_consensus.cc:504] T 2e5de8763ff94675953913c917b16d68 P aad2e8806cab4a71bcdc7e7372c36854 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } } peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } }
I20260926 20:49:19.328357 8422 leader_election.cc:288] T 2e5de8763ff94675953913c917b16d68 P aad2e8806cab4a71bcdc7e7372c36854 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ebea5fcb4064638863a64bdbf8bb189 (127.0.98.2:43131), b7762c6bd65f4238ac92779f523d6b1c (127.0.98.1:38323)
W20260926 20:49:19.328760 8248 tablet.cc:2369] T 2e5de8763ff94675953913c917b16d68 P 5ebea5fcb4064638863a64bdbf8bb189: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:49:19.331334 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:19.328709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53995 (local address 127.0.98.2:43131)
0926 20:49:19.328783 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:19.328785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:19.328795 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:49:19.328821 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:19.328822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:19.328835 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:19.328926 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:49:19.329081 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.329630 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.330592 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.330992 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.331045 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:19.331053 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:19.331227 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 20:49:19.331268 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 20:49:19.331363 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:19.328879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43427 (local address 127.0.98.1:38323)
0926 20:49:19.328991 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:19.328993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:19.329000 (+ 7us) server_negotiation.cc:407] Connection header received
0926 20:49:19.329041 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:19.329042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:19.329054 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:19.329116 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:49:19.329269 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.329744 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.330662 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.331006 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.331099 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:19.331109 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:19.331267 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 20:49:19.331302 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 20:49:19.331357 8280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:19.328583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43131 (local address 127.0.98.3:53995)
0926 20:49:19.328663 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:19.328677 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:19.328729 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:19.328928 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:19.328929 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:19.328931 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:49:19.329002 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:19.329005 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.329705 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.329706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:19.330496 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:19.330498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.331064 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.331065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:19.331155 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:19.331158 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:49:19.331209 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 20:49:19.331271 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 20:49:19.331427 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:19.328690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38323 (local address 127.0.98.3:43427)
0926 20:49:19.328860 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:19.328870 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:19.328914 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:19.329125 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:19.329126 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:19.329128 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:49:19.329199 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:19.329200 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.329838 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.329839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:19.330574 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:19.330576 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.331119 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.331120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:19.331171 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:19.331174 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:49:19.331247 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 20:49:19.331293 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":11,"thread_start_us":113,"threads_started":1}
I20260926 20:49:19.331739 8061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5de8763ff94675953913c917b16d68" candidate_uuid: "aad2e8806cab4a71bcdc7e7372c36854" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7762c6bd65f4238ac92779f523d6b1c" is_pre_election: true
I20260926 20:49:19.331790 8201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5de8763ff94675953913c917b16d68" candidate_uuid: "aad2e8806cab4a71bcdc7e7372c36854" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ebea5fcb4064638863a64bdbf8bb189" is_pre_election: true
I20260926 20:49:19.331923 8061 raft_consensus.cc:2454] T 2e5de8763ff94675953913c917b16d68 P b7762c6bd65f4238ac92779f523d6b1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad2e8806cab4a71bcdc7e7372c36854 in term 0.
I20260926 20:49:19.331941 8201 raft_consensus.cc:2454] T 2e5de8763ff94675953913c917b16d68 P 5ebea5fcb4064638863a64bdbf8bb189 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad2e8806cab4a71bcdc7e7372c36854 in term 0.
I20260926 20:49:19.332152 8272 leader_election.cc:302] T 2e5de8763ff94675953913c917b16d68 P aad2e8806cab4a71bcdc7e7372c36854 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ebea5fcb4064638863a64bdbf8bb189, aad2e8806cab4a71bcdc7e7372c36854; no voters:
I20260926 20:49:19.332291 8422 raft_consensus.cc:2781] T 2e5de8763ff94675953913c917b16d68 P aad2e8806cab4a71bcdc7e7372c36854 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:49:19.332342 8422 raft_consensus.cc:482] T 2e5de8763ff94675953913c917b16d68 P aad2e8806cab4a71bcdc7e7372c36854 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:49:19.332368 8422 raft_consensus.cc:3037] T 2e5de8763ff94675953913c917b16d68 P aad2e8806cab4a71bcdc7e7372c36854 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:49:19.333104 8422 raft_consensus.cc:504] T 2e5de8763ff94675953913c917b16d68 P aad2e8806cab4a71bcdc7e7372c36854 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } } peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } }
I20260926 20:49:19.333226 8422 leader_election.cc:288] T 2e5de8763ff94675953913c917b16d68 P aad2e8806cab4a71bcdc7e7372c36854 [CANDIDATE]: Term 1 election: Requested vote from peers 5ebea5fcb4064638863a64bdbf8bb189 (127.0.98.2:43131), b7762c6bd65f4238ac92779f523d6b1c (127.0.98.1:38323)
I20260926 20:49:19.333412 8061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5de8763ff94675953913c917b16d68" candidate_uuid: "aad2e8806cab4a71bcdc7e7372c36854" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7762c6bd65f4238ac92779f523d6b1c"
I20260926 20:49:19.333480 8061 raft_consensus.cc:3037] T 2e5de8763ff94675953913c917b16d68 P b7762c6bd65f4238ac92779f523d6b1c [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:49:19.333464 8201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5de8763ff94675953913c917b16d68" candidate_uuid: "aad2e8806cab4a71bcdc7e7372c36854" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ebea5fcb4064638863a64bdbf8bb189"
I20260926 20:49:19.333567 8201 raft_consensus.cc:3037] T 2e5de8763ff94675953913c917b16d68 P 5ebea5fcb4064638863a64bdbf8bb189 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:49:19.334154 8061 raft_consensus.cc:2454] T 2e5de8763ff94675953913c917b16d68 P b7762c6bd65f4238ac92779f523d6b1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad2e8806cab4a71bcdc7e7372c36854 in term 1.
I20260926 20:49:19.334285 8201 raft_consensus.cc:2454] T 2e5de8763ff94675953913c917b16d68 P 5ebea5fcb4064638863a64bdbf8bb189 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad2e8806cab4a71bcdc7e7372c36854 in term 1.
I20260926 20:49:19.334311 8272 leader_election.cc:302] T 2e5de8763ff94675953913c917b16d68 P aad2e8806cab4a71bcdc7e7372c36854 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aad2e8806cab4a71bcdc7e7372c36854, b7762c6bd65f4238ac92779f523d6b1c; no voters:
I20260926 20:49:19.334388 8422 raft_consensus.cc:2781] T 2e5de8763ff94675953913c917b16d68 P aad2e8806cab4a71bcdc7e7372c36854 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:49:19.334528 8422 raft_consensus.cc:686] T 2e5de8763ff94675953913c917b16d68 P aad2e8806cab4a71bcdc7e7372c36854 [term 1 LEADER]: Becoming Leader. State: Replica: aad2e8806cab4a71bcdc7e7372c36854, State: Running, Role: LEADER
I20260926 20:49:19.334632 8422 consensus_queue.cc:237] T 2e5de8763ff94675953913c917b16d68 P aad2e8806cab4a71bcdc7e7372c36854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } } peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } }
I20260926 20:49:19.335470 7903 catalog_manager.cc:5495] T 2e5de8763ff94675953913c917b16d68 P aad2e8806cab4a71bcdc7e7372c36854 reported cstate change: term changed from 0 to 1, leader changed from <none> to aad2e8806cab4a71bcdc7e7372c36854 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "aad2e8806cab4a71bcdc7e7372c36854" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } health_report { overall_health: HEALTHY } } }
I20260926 20:49:19.346109 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:19.343850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33172 (local address 127.0.98.3:37051)
0926 20:49:19.343911 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:19.343914 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:19.343922 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:49:19.343988 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:19.343989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:19.344006 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:19.344062 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:49:19.344186 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.344665 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.345442 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.345633 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.345688 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:19.345817 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:49:19.345903 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:49:19.345958 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:19.345988 (+ 30us) server_negotiation.cc:299] Negotiation successful
0926 20:49:19.346004 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 20:49:19.347950 8061 raft_consensus.cc:1261] T 2e5de8763ff94675953913c917b16d68 P b7762c6bd65f4238ac92779f523d6b1c [term 1 FOLLOWER]: Refusing update from remote peer aad2e8806cab4a71bcdc7e7372c36854: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:49:19.347957 8201 raft_consensus.cc:1261] T 2e5de8763ff94675953913c917b16d68 P 5ebea5fcb4064638863a64bdbf8bb189 [term 1 FOLLOWER]: Refusing update from remote peer aad2e8806cab4a71bcdc7e7372c36854: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:49:19.348266 8425 consensus_queue.cc:1035] T 2e5de8763ff94675953913c917b16d68 P aad2e8806cab4a71bcdc7e7372c36854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:49:19.348387 8422 consensus_queue.cc:1035] T 2e5de8763ff94675953913c917b16d68 P aad2e8806cab4a71bcdc7e7372c36854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:49:19.350811 8429 mvcc.cc:205] Tried to move back new op lower bound from 7333706790286557184 to 7333706790235295744. Current Snapshot: MvccSnapshot[applied={T|T < 7333706790286557184}]
I20260926 20:49:19.353395 7904 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:49:19.353472 7904 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:49:19.354228 7904 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b012d088b8243c290a80ca96476dd0c: Sending DeleteTablet for 3 replicas of tablet 2e5de8763ff94675953913c917b16d68
I20260926 20:49:19.354538 8181 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e5de8763ff94675953913c917b16d68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:49:19 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48720
I20260926 20:49:19.354575 8321 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e5de8763ff94675953913c917b16d68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:49:19 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33166
I20260926 20:49:19.354592 8041 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e5de8763ff94675953913c917b16d68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:49:19 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53438
I20260926 20:49:19.354859 8440 tablet_replica.cc:330] T 2e5de8763ff94675953913c917b16d68 P b7762c6bd65f4238ac92779f523d6b1c: stopping tablet replica
I20260926 20:49:19.354971 8438 tablet_replica.cc:330] T 2e5de8763ff94675953913c917b16d68 P 5ebea5fcb4064638863a64bdbf8bb189: stopping tablet replica
I20260926 20:49:19.354990 8440 raft_consensus.cc:2229] T 2e5de8763ff94675953913c917b16d68 P b7762c6bd65f4238ac92779f523d6b1c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:49:19.355077 8438 raft_consensus.cc:2229] T 2e5de8763ff94675953913c917b16d68 P 5ebea5fcb4064638863a64bdbf8bb189 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:49:19.355049 7904 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45862:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 20:49:19.355187 8438 raft_consensus.cc:2258] T 2e5de8763ff94675953913c917b16d68 P 5ebea5fcb4064638863a64bdbf8bb189 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 20:49:19.355185 7904 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:49:19.355427 8440 raft_consensus.cc:2258] T 2e5de8763ff94675953913c917b16d68 P b7762c6bd65f4238ac92779f523d6b1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:49:19.356279 8438 ts_tablet_manager.cc:1905] T 2e5de8763ff94675953913c917b16d68 P 5ebea5fcb4064638863a64bdbf8bb189: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:49:19.356392 8181 tablet_service.cc:1467] Processing CreateTablet for tablet 50dab45176a5469fbdc9452774754c9c (DEFAULT_TABLE table=test-table [id=7da1dec31b91453f8466227f3718565a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:49:19.356392 8321 tablet_service.cc:1467] Processing CreateTablet for tablet 50dab45176a5469fbdc9452774754c9c (DEFAULT_TABLE table=test-table [id=7da1dec31b91453f8466227f3718565a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:49:19.356493 8321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50dab45176a5469fbdc9452774754c9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:49:19.356493 8181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50dab45176a5469fbdc9452774754c9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:49:19.357751 8418 tablet_bootstrap.cc:492] T 50dab45176a5469fbdc9452774754c9c P aad2e8806cab4a71bcdc7e7372c36854: Bootstrap starting.
I20260926 20:49:19.357765 8415 tablet_bootstrap.cc:492] T 50dab45176a5469fbdc9452774754c9c P 5ebea5fcb4064638863a64bdbf8bb189: Bootstrap starting.
I20260926 20:49:19.355923 8440 ts_tablet_manager.cc:1905] T 2e5de8763ff94675953913c917b16d68 P b7762c6bd65f4238ac92779f523d6b1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:49:19.358157 8418 tablet_bootstrap.cc:654] T 50dab45176a5469fbdc9452774754c9c P aad2e8806cab4a71bcdc7e7372c36854: Neither blocks nor log segments found. Creating new log.
I20260926 20:49:19.358170 8415 tablet_bootstrap.cc:654] T 50dab45176a5469fbdc9452774754c9c P 5ebea5fcb4064638863a64bdbf8bb189: Neither blocks nor log segments found. Creating new log.
I20260926 20:49:19.358750 8438 ts_tablet_manager.cc:1918] T 2e5de8763ff94675953913c917b16d68 P 5ebea5fcb4064638863a64bdbf8bb189: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:49:19.358821 8438 log.cc:1196] T 2e5de8763ff94675953913c917b16d68 P 5ebea5fcb4064638863a64bdbf8bb189: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-1/wal/wals/2e5de8763ff94675953913c917b16d68
I20260926 20:49:19.358911 8041 tablet_service.cc:1467] Processing CreateTablet for tablet 50dab45176a5469fbdc9452774754c9c (DEFAULT_TABLE table=test-table [id=7da1dec31b91453f8466227f3718565a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:49:19.359009 8041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50dab45176a5469fbdc9452774754c9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:49:19.359078 8438 ts_tablet_manager.cc:1939] T 2e5de8763ff94675953913c917b16d68 P 5ebea5fcb4064638863a64bdbf8bb189: Deleting consensus metadata
I20260926 20:49:19.359637 7888 catalog_manager.cc:4841] TS 5ebea5fcb4064638863a64bdbf8bb189 (127.0.98.2:43131): tablet 2e5de8763ff94675953913c917b16d68 (table test-table [id=2ddfe76eeec54521bebe4315052f16ce]) successfully deleted
I20260926 20:49:19.359800 8440 ts_tablet_manager.cc:1918] T 2e5de8763ff94675953913c917b16d68 P b7762c6bd65f4238ac92779f523d6b1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:49:19.359858 8440 log.cc:1196] T 2e5de8763ff94675953913c917b16d68 P b7762c6bd65f4238ac92779f523d6b1c: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-0/wal/wals/2e5de8763ff94675953913c917b16d68
I20260926 20:49:19.359954 8418 tablet_bootstrap.cc:492] T 50dab45176a5469fbdc9452774754c9c P aad2e8806cab4a71bcdc7e7372c36854: No bootstrap required, opened a new log
I20260926 20:49:19.359999 8418 ts_tablet_manager.cc:1397] T 50dab45176a5469fbdc9452774754c9c P aad2e8806cab4a71bcdc7e7372c36854: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:49:19.360103 8440 ts_tablet_manager.cc:1939] T 2e5de8763ff94675953913c917b16d68 P b7762c6bd65f4238ac92779f523d6b1c: Deleting consensus metadata
I20260926 20:49:19.360132 8418 raft_consensus.cc:348] T 50dab45176a5469fbdc9452774754c9c P aad2e8806cab4a71bcdc7e7372c36854 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } } peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } }
I20260926 20:49:19.360194 8418 raft_consensus.cc:374] T 50dab45176a5469fbdc9452774754c9c P aad2e8806cab4a71bcdc7e7372c36854 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:49:19.360216 8418 raft_consensus.cc:729] T 50dab45176a5469fbdc9452774754c9c P aad2e8806cab4a71bcdc7e7372c36854 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad2e8806cab4a71bcdc7e7372c36854, State: Initialized, Role: FOLLOWER
I20260926 20:49:19.360260 8418 consensus_queue.cc:260] T 50dab45176a5469fbdc9452774754c9c P aad2e8806cab4a71bcdc7e7372c36854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } } peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } }
I20260926 20:49:19.360337 8416 tablet_bootstrap.cc:492] T 50dab45176a5469fbdc9452774754c9c P b7762c6bd65f4238ac92779f523d6b1c: Bootstrap starting.
I20260926 20:49:19.360363 8418 ts_tablet_manager.cc:1428] T 50dab45176a5469fbdc9452774754c9c P aad2e8806cab4a71bcdc7e7372c36854: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:19.360718 8416 tablet_bootstrap.cc:654] T 50dab45176a5469fbdc9452774754c9c P b7762c6bd65f4238ac92779f523d6b1c: Neither blocks nor log segments found. Creating new log.
I20260926 20:49:19.361094 7888 catalog_manager.cc:4841] TS b7762c6bd65f4238ac92779f523d6b1c (127.0.98.1:38323): tablet 2e5de8763ff94675953913c917b16d68 (table test-table [id=2ddfe76eeec54521bebe4315052f16ce]) successfully deleted
I20260926 20:49:19.361253 8416 tablet_bootstrap.cc:492] T 50dab45176a5469fbdc9452774754c9c P b7762c6bd65f4238ac92779f523d6b1c: No bootstrap required, opened a new log
I20260926 20:49:19.361300 8416 ts_tablet_manager.cc:1397] T 50dab45176a5469fbdc9452774754c9c P b7762c6bd65f4238ac92779f523d6b1c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 20:49:19.361441 8416 raft_consensus.cc:348] T 50dab45176a5469fbdc9452774754c9c P b7762c6bd65f4238ac92779f523d6b1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } } peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } }
I20260926 20:49:19.361529 8416 raft_consensus.cc:374] T 50dab45176a5469fbdc9452774754c9c P b7762c6bd65f4238ac92779f523d6b1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:49:19.361557 8416 raft_consensus.cc:729] T 50dab45176a5469fbdc9452774754c9c P b7762c6bd65f4238ac92779f523d6b1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7762c6bd65f4238ac92779f523d6b1c, State: Initialized, Role: FOLLOWER
I20260926 20:49:19.361608 8416 consensus_queue.cc:260] T 50dab45176a5469fbdc9452774754c9c P b7762c6bd65f4238ac92779f523d6b1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } } peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } }
I20260926 20:49:19.361713 8416 ts_tablet_manager.cc:1428] T 50dab45176a5469fbdc9452774754c9c P b7762c6bd65f4238ac92779f523d6b1c: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 20:49:19.361907 8415 tablet_bootstrap.cc:492] T 50dab45176a5469fbdc9452774754c9c P 5ebea5fcb4064638863a64bdbf8bb189: No bootstrap required, opened a new log
I20260926 20:49:19.361950 8415 ts_tablet_manager.cc:1397] T 50dab45176a5469fbdc9452774754c9c P 5ebea5fcb4064638863a64bdbf8bb189: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 20:49:19.362080 8415 raft_consensus.cc:348] T 50dab45176a5469fbdc9452774754c9c P 5ebea5fcb4064638863a64bdbf8bb189 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } } peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } }
I20260926 20:49:19.362141 8415 raft_consensus.cc:374] T 50dab45176a5469fbdc9452774754c9c P 5ebea5fcb4064638863a64bdbf8bb189 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:49:19.362162 8415 raft_consensus.cc:729] T 50dab45176a5469fbdc9452774754c9c P 5ebea5fcb4064638863a64bdbf8bb189 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ebea5fcb4064638863a64bdbf8bb189, State: Initialized, Role: FOLLOWER
I20260926 20:49:19.362219 8415 consensus_queue.cc:260] T 50dab45176a5469fbdc9452774754c9c P 5ebea5fcb4064638863a64bdbf8bb189 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } } peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } }
I20260926 20:49:19.362298 8415 ts_tablet_manager.cc:1428] T 50dab45176a5469fbdc9452774754c9c P 5ebea5fcb4064638863a64bdbf8bb189: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 20:49:19.362488 8439 tablet_replica.cc:330] T 2e5de8763ff94675953913c917b16d68 P aad2e8806cab4a71bcdc7e7372c36854: stopping tablet replica
I20260926 20:49:19.362594 8439 raft_consensus.cc:2229] T 2e5de8763ff94675953913c917b16d68 P aad2e8806cab4a71bcdc7e7372c36854 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:49:19.362726 8439 raft_consensus.cc:2258] T 2e5de8763ff94675953913c917b16d68 P aad2e8806cab4a71bcdc7e7372c36854 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:49:19.364041 8439 ts_tablet_manager.cc:1905] T 2e5de8763ff94675953913c917b16d68 P aad2e8806cab4a71bcdc7e7372c36854: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:49:19.365392 8439 ts_tablet_manager.cc:1918] T 2e5de8763ff94675953913c917b16d68 P aad2e8806cab4a71bcdc7e7372c36854: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:49:19.365466 8439 log.cc:1196] T 2e5de8763ff94675953913c917b16d68 P aad2e8806cab4a71bcdc7e7372c36854: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-2/wal/wals/2e5de8763ff94675953913c917b16d68
I20260926 20:49:19.365489 8421 raft_consensus.cc:482] T 50dab45176a5469fbdc9452774754c9c P 5ebea5fcb4064638863a64bdbf8bb189 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:49:19.365567 8421 raft_consensus.cc:504] T 50dab45176a5469fbdc9452774754c9c P 5ebea5fcb4064638863a64bdbf8bb189 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } } peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } }
I20260926 20:49:19.365762 8439 ts_tablet_manager.cc:1939] T 2e5de8763ff94675953913c917b16d68 P aad2e8806cab4a71bcdc7e7372c36854: Deleting consensus metadata
I20260926 20:49:19.365830 8421 leader_election.cc:288] T 50dab45176a5469fbdc9452774754c9c P 5ebea5fcb4064638863a64bdbf8bb189 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7762c6bd65f4238ac92779f523d6b1c (127.0.98.1:38323), aad2e8806cab4a71bcdc7e7372c36854 (127.0.98.3:37051)
I20260926 20:49:19.367561 7889 catalog_manager.cc:4841] TS aad2e8806cab4a71bcdc7e7372c36854 (127.0.98.3:37051): tablet 2e5de8763ff94675953913c917b16d68 (table test-table [id=2ddfe76eeec54521bebe4315052f16ce]) successfully deleted
I20260926 20:49:19.368844 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:19.366063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47519 (local address 127.0.98.3:37051)
0926 20:49:19.366116 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:19.366118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:19.366128 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:49:19.366191 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:19.366193 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:19.366202 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:19.366249 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:49:19.366431 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.367032 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.368141 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.368507 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.368590 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:19.368599 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:19.368751 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 20:49:19.368784 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 20:49:19.368868 8140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:19.365971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37051 (local address 127.0.98.2:47519)
0926 20:49:19.366074 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:19.366085 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:19.366122 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:19.366295 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:19.366295 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:19.366297 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:49:19.366359 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:19.366361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.367121 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.367122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:19.368059 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:19.368061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.368588 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.368589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:19.368660 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:19.368662 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 20:49:19.368744 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 20:49:19.368790 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260926 20:49:19.369102 8341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50dab45176a5469fbdc9452774754c9c" candidate_uuid: "5ebea5fcb4064638863a64bdbf8bb189" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad2e8806cab4a71bcdc7e7372c36854" is_pre_election: true
I20260926 20:49:19.369191 8341 raft_consensus.cc:2454] T 50dab45176a5469fbdc9452774754c9c P aad2e8806cab4a71bcdc7e7372c36854 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ebea5fcb4064638863a64bdbf8bb189 in term 0.
I20260926 20:49:19.369338 8133 leader_election.cc:302] T 50dab45176a5469fbdc9452774754c9c P 5ebea5fcb4064638863a64bdbf8bb189 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ebea5fcb4064638863a64bdbf8bb189, aad2e8806cab4a71bcdc7e7372c36854; no voters:
I20260926 20:49:19.369658 8421 raft_consensus.cc:2781] T 50dab45176a5469fbdc9452774754c9c P 5ebea5fcb4064638863a64bdbf8bb189 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:49:19.369712 8421 raft_consensus.cc:482] T 50dab45176a5469fbdc9452774754c9c P 5ebea5fcb4064638863a64bdbf8bb189 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:49:19.369733 8421 raft_consensus.cc:3037] T 50dab45176a5469fbdc9452774754c9c P 5ebea5fcb4064638863a64bdbf8bb189 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:49:19.370508 8421 raft_consensus.cc:504] T 50dab45176a5469fbdc9452774754c9c P 5ebea5fcb4064638863a64bdbf8bb189 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } } peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } }
I20260926 20:49:19.370641 8421 leader_election.cc:288] T 50dab45176a5469fbdc9452774754c9c P 5ebea5fcb4064638863a64bdbf8bb189 [CANDIDATE]: Term 1 election: Requested vote from peers b7762c6bd65f4238ac92779f523d6b1c (127.0.98.1:38323), aad2e8806cab4a71bcdc7e7372c36854 (127.0.98.3:37051)
I20260926 20:49:19.370895 8341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50dab45176a5469fbdc9452774754c9c" candidate_uuid: "5ebea5fcb4064638863a64bdbf8bb189" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad2e8806cab4a71bcdc7e7372c36854"
I20260926 20:49:19.370972 8341 raft_consensus.cc:3037] T 50dab45176a5469fbdc9452774754c9c P aad2e8806cab4a71bcdc7e7372c36854 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:49:19.371083 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:19.366242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59479 (local address 127.0.98.1:38323)
0926 20:49:19.366281 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:19.366282 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:19.366289 (+ 7us) server_negotiation.cc:407] Connection header received
0926 20:49:19.366310 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:19.366311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:19.366321 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:19.366374 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:49:19.366673 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.367405 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.368920 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.369517 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.369570 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:19.369578 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:19.370967 (+ 1389us) server_negotiation.cc:299] Negotiation successful
0926 20:49:19.371010 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 20:49:19.371271 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:19.366009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38323 (local address 127.0.98.2:59479)
0926 20:49:19.366210 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:19.366219 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:19.366266 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:19.366377 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:19.366378 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:19.366379 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:49:19.366586 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:19.366589 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.367569 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.367570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:19.368815 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:19.368818 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.370804 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.370805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:19.370884 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:19.370887 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:49:19.371154 (+ 267us) client_negotiation.cc:240] Negotiation successful
0926 20:49:19.371197 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":45,"threads_started":1}
I20260926 20:49:19.371537 8061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50dab45176a5469fbdc9452774754c9c" candidate_uuid: "5ebea5fcb4064638863a64bdbf8bb189" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7762c6bd65f4238ac92779f523d6b1c" is_pre_election: true
I20260926 20:49:19.371618 8061 raft_consensus.cc:2454] T 50dab45176a5469fbdc9452774754c9c P b7762c6bd65f4238ac92779f523d6b1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ebea5fcb4064638863a64bdbf8bb189 in term 0.
I20260926 20:49:19.371577 8060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50dab45176a5469fbdc9452774754c9c" candidate_uuid: "5ebea5fcb4064638863a64bdbf8bb189" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7762c6bd65f4238ac92779f523d6b1c"
I20260926 20:49:19.371655 8060 raft_consensus.cc:3037] T 50dab45176a5469fbdc9452774754c9c P b7762c6bd65f4238ac92779f523d6b1c [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:49:19.371832 8341 raft_consensus.cc:2454] T 50dab45176a5469fbdc9452774754c9c P aad2e8806cab4a71bcdc7e7372c36854 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ebea5fcb4064638863a64bdbf8bb189 in term 1.
I20260926 20:49:19.372337 8060 raft_consensus.cc:2454] T 50dab45176a5469fbdc9452774754c9c P b7762c6bd65f4238ac92779f523d6b1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ebea5fcb4064638863a64bdbf8bb189 in term 1.
I20260926 20:49:19.372553 8132 leader_election.cc:302] T 50dab45176a5469fbdc9452774754c9c P 5ebea5fcb4064638863a64bdbf8bb189 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ebea5fcb4064638863a64bdbf8bb189, b7762c6bd65f4238ac92779f523d6b1c; no voters:
I20260926 20:49:19.372635 8421 raft_consensus.cc:2781] T 50dab45176a5469fbdc9452774754c9c P 5ebea5fcb4064638863a64bdbf8bb189 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:49:19.372773 8421 raft_consensus.cc:686] T 50dab45176a5469fbdc9452774754c9c P 5ebea5fcb4064638863a64bdbf8bb189 [term 1 LEADER]: Becoming Leader. State: Replica: 5ebea5fcb4064638863a64bdbf8bb189, State: Running, Role: LEADER
I20260926 20:49:19.372838 8421 consensus_queue.cc:237] T 50dab45176a5469fbdc9452774754c9c P 5ebea5fcb4064638863a64bdbf8bb189 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } } peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } }
I20260926 20:49:19.373383 7904 catalog_manager.cc:5495] T 50dab45176a5469fbdc9452774754c9c P 5ebea5fcb4064638863a64bdbf8bb189 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ebea5fcb4064638863a64bdbf8bb189 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "5ebea5fcb4064638863a64bdbf8bb189" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } health_report { overall_health: UNKNOWN } } }
Sep 26 20:49:19 dist-test-slave-hwlj krb5kdc[7859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455758, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 20:49:19.408208 8452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:19.400482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32897 (local address 127.0.0.1:45878)
0926 20:49:19.400638 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:19.400655 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:19.400713 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:19.400857 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:19.400858 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:19.401152 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:19.401205 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:19.401207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.402157 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.402158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:19.402755 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:19.402757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.403197 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.403198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:19.403240 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:19.403839 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:19.403848 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:19.405294 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:19.406869 (+ 1575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:19.406877 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:19.406880 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:19.407147 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:19.407416 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:19.407417 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:19.407418 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:19.407488 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:19.407778 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:19.407797 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:19.407853 (+ 56us) client_negotiation.cc:769] Sending connection context
0926 20:49:19.407925 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 20:49:19.408049 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":93,"threads_started":1}
I20260926 20:49:19.408313 7972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:19.400549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45878 (local address 127.0.98.62:32897)
0926 20:49:19.400607 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:19.400609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:19.400719 (+ 110us) server_negotiation.cc:407] Connection header received
0926 20:49:19.400753 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:19.400754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:19.400767 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:19.400845 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:19.401281 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.402065 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.402837 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.403117 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.403191 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:19.405405 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:19.405407 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:19.405407 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:19.405413 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:19.406786 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:19.407273 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:19.407275 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:19.407275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:19.407343 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:19.407587 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:19.407588 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:19.407588 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:19.407699 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:19.407797 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:19.407961 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 20:49:19.408101 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":35}
I20260926 20:49:19.410365 7904 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45878:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 20:49:19.410594 7904 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:49:19.412149 8041 tablet_service.cc:1467] Processing CreateTablet for tablet 2aede59f334b4b8fa9bd69d4211c4d16 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2cc70c6d4d704232a1c35a05c4d19a03]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:49:19.412259 8181 tablet_service.cc:1467] Processing CreateTablet for tablet 2aede59f334b4b8fa9bd69d4211c4d16 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2cc70c6d4d704232a1c35a05c4d19a03]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:49:19.412393 8041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2aede59f334b4b8fa9bd69d4211c4d16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:49:19.412410 8181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2aede59f334b4b8fa9bd69d4211c4d16. 1 dirs total, 0 dirs full, 0 dirs failed
W20260926 20:49:19.412775 8108 tablet.cc:2369] T 50dab45176a5469fbdc9452774754c9c P b7762c6bd65f4238ac92779f523d6b1c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:49:19.413798 8415 tablet_bootstrap.cc:492] T 2aede59f334b4b8fa9bd69d4211c4d16 P 5ebea5fcb4064638863a64bdbf8bb189: Bootstrap starting.
I20260926 20:49:19.413898 8321 tablet_service.cc:1467] Processing CreateTablet for tablet 2aede59f334b4b8fa9bd69d4211c4d16 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2cc70c6d4d704232a1c35a05c4d19a03]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:49:19.414027 8321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2aede59f334b4b8fa9bd69d4211c4d16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:49:19.414206 8415 tablet_bootstrap.cc:654] T 2aede59f334b4b8fa9bd69d4211c4d16 P 5ebea5fcb4064638863a64bdbf8bb189: Neither blocks nor log segments found. Creating new log.
I20260926 20:49:19.415150 8415 tablet_bootstrap.cc:492] T 2aede59f334b4b8fa9bd69d4211c4d16 P 5ebea5fcb4064638863a64bdbf8bb189: No bootstrap required, opened a new log
I20260926 20:49:19.415210 8415 ts_tablet_manager.cc:1397] T 2aede59f334b4b8fa9bd69d4211c4d16 P 5ebea5fcb4064638863a64bdbf8bb189: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:49:19.415375 8415 raft_consensus.cc:348] T 2aede59f334b4b8fa9bd69d4211c4d16 P 5ebea5fcb4064638863a64bdbf8bb189 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } } peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } }
I20260926 20:49:19.415467 8415 raft_consensus.cc:374] T 2aede59f334b4b8fa9bd69d4211c4d16 P 5ebea5fcb4064638863a64bdbf8bb189 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:49:19.415488 8415 raft_consensus.cc:729] T 2aede59f334b4b8fa9bd69d4211c4d16 P 5ebea5fcb4064638863a64bdbf8bb189 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ebea5fcb4064638863a64bdbf8bb189, State: Initialized, Role: FOLLOWER
I20260926 20:49:19.415529 8415 consensus_queue.cc:260] T 2aede59f334b4b8fa9bd69d4211c4d16 P 5ebea5fcb4064638863a64bdbf8bb189 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } } peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } }
I20260926 20:49:19.415581 8416 tablet_bootstrap.cc:492] T 2aede59f334b4b8fa9bd69d4211c4d16 P b7762c6bd65f4238ac92779f523d6b1c: Bootstrap starting.
I20260926 20:49:19.415627 8415 ts_tablet_manager.cc:1428] T 2aede59f334b4b8fa9bd69d4211c4d16 P 5ebea5fcb4064638863a64bdbf8bb189: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 20:49:19.415733 8421 tablet_replica.cc:439] T 2aede59f334b4b8fa9bd69d4211c4d16 P 5ebea5fcb4064638863a64bdbf8bb189: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } } peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } } }
I20260926 20:49:19.415848 8421 tablet_replica.cc:442] T 2aede59f334b4b8fa9bd69d4211c4d16 P 5ebea5fcb4064638863a64bdbf8bb189: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:49:19.416044 8416 tablet_bootstrap.cc:654] T 2aede59f334b4b8fa9bd69d4211c4d16 P b7762c6bd65f4238ac92779f523d6b1c: Neither blocks nor log segments found. Creating new log.
I20260926 20:49:19.416388 8418 tablet_bootstrap.cc:492] T 2aede59f334b4b8fa9bd69d4211c4d16 P aad2e8806cab4a71bcdc7e7372c36854: Bootstrap starting.
I20260926 20:49:19.416719 8418 tablet_bootstrap.cc:654] T 2aede59f334b4b8fa9bd69d4211c4d16 P aad2e8806cab4a71bcdc7e7372c36854: Neither blocks nor log segments found. Creating new log.
I20260926 20:49:19.417351 8416 tablet_bootstrap.cc:492] T 2aede59f334b4b8fa9bd69d4211c4d16 P b7762c6bd65f4238ac92779f523d6b1c: No bootstrap required, opened a new log
I20260926 20:49:19.417392 8418 tablet_bootstrap.cc:492] T 2aede59f334b4b8fa9bd69d4211c4d16 P aad2e8806cab4a71bcdc7e7372c36854: No bootstrap required, opened a new log
I20260926 20:49:19.417407 8416 ts_tablet_manager.cc:1397] T 2aede59f334b4b8fa9bd69d4211c4d16 P b7762c6bd65f4238ac92779f523d6b1c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 20:49:19.417430 8418 ts_tablet_manager.cc:1397] T 2aede59f334b4b8fa9bd69d4211c4d16 P aad2e8806cab4a71bcdc7e7372c36854: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:49:19.417559 8418 raft_consensus.cc:348] T 2aede59f334b4b8fa9bd69d4211c4d16 P aad2e8806cab4a71bcdc7e7372c36854 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } } peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } }
I20260926 20:49:19.417559 8416 raft_consensus.cc:348] T 2aede59f334b4b8fa9bd69d4211c4d16 P b7762c6bd65f4238ac92779f523d6b1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } } peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } }
I20260926 20:49:19.417629 8416 raft_consensus.cc:374] T 2aede59f334b4b8fa9bd69d4211c4d16 P b7762c6bd65f4238ac92779f523d6b1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:49:19.417629 8418 raft_consensus.cc:374] T 2aede59f334b4b8fa9bd69d4211c4d16 P aad2e8806cab4a71bcdc7e7372c36854 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:49:19.417654 8418 raft_consensus.cc:729] T 2aede59f334b4b8fa9bd69d4211c4d16 P aad2e8806cab4a71bcdc7e7372c36854 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad2e8806cab4a71bcdc7e7372c36854, State: Initialized, Role: FOLLOWER
I20260926 20:49:19.417654 8416 raft_consensus.cc:729] T 2aede59f334b4b8fa9bd69d4211c4d16 P b7762c6bd65f4238ac92779f523d6b1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7762c6bd65f4238ac92779f523d6b1c, State: Initialized, Role: FOLLOWER
I20260926 20:49:19.417695 8416 consensus_queue.cc:260] T 2aede59f334b4b8fa9bd69d4211c4d16 P b7762c6bd65f4238ac92779f523d6b1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } } peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } }
I20260926 20:49:19.417692 8418 consensus_queue.cc:260] T 2aede59f334b4b8fa9bd69d4211c4d16 P aad2e8806cab4a71bcdc7e7372c36854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } } peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } }
I20260926 20:49:19.417785 8418 ts_tablet_manager.cc:1428] T 2aede59f334b4b8fa9bd69d4211c4d16 P aad2e8806cab4a71bcdc7e7372c36854: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:19.417785 8416 ts_tablet_manager.cc:1428] T 2aede59f334b4b8fa9bd69d4211c4d16 P b7762c6bd65f4238ac92779f523d6b1c: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 20:49:19.417948 8423 tablet_replica.cc:439] T 2aede59f334b4b8fa9bd69d4211c4d16 P b7762c6bd65f4238ac92779f523d6b1c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } } peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } } }
I20260926 20:49:19.418025 8423 tablet_replica.cc:442] T 2aede59f334b4b8fa9bd69d4211c4d16 P b7762c6bd65f4238ac92779f523d6b1c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:49:19.418120 8433 tablet_replica.cc:439] T 2aede59f334b4b8fa9bd69d4211c4d16 P aad2e8806cab4a71bcdc7e7372c36854: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } } peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } } }
I20260926 20:49:19.418192 8433 tablet_replica.cc:442] T 2aede59f334b4b8fa9bd69d4211c4d16 P aad2e8806cab4a71bcdc7e7372c36854: This TxnStatusTablet replica's current role is: FOLLOWER
W20260926 20:49:19.499013 8388 tablet.cc:2369] T 2aede59f334b4b8fa9bd69d4211c4d16 P aad2e8806cab4a71bcdc7e7372c36854: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:49:19.560423 8433 raft_consensus.cc:482] T 2aede59f334b4b8fa9bd69d4211c4d16 P aad2e8806cab4a71bcdc7e7372c36854 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:49:19.560523 8433 raft_consensus.cc:504] T 2aede59f334b4b8fa9bd69d4211c4d16 P aad2e8806cab4a71bcdc7e7372c36854 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } } peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } }
I20260926 20:49:19.560668 8433 leader_election.cc:288] T 2aede59f334b4b8fa9bd69d4211c4d16 P aad2e8806cab4a71bcdc7e7372c36854 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7762c6bd65f4238ac92779f523d6b1c (127.0.98.1:38323), 5ebea5fcb4064638863a64bdbf8bb189 (127.0.98.2:43131)
I20260926 20:49:19.560895 8060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aede59f334b4b8fa9bd69d4211c4d16" candidate_uuid: "aad2e8806cab4a71bcdc7e7372c36854" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7762c6bd65f4238ac92779f523d6b1c" is_pre_election: true
I20260926 20:49:19.560986 8060 raft_consensus.cc:2454] T 2aede59f334b4b8fa9bd69d4211c4d16 P b7762c6bd65f4238ac92779f523d6b1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad2e8806cab4a71bcdc7e7372c36854 in term 0.
I20260926 20:49:19.560905 8201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aede59f334b4b8fa9bd69d4211c4d16" candidate_uuid: "aad2e8806cab4a71bcdc7e7372c36854" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ebea5fcb4064638863a64bdbf8bb189" is_pre_election: true
I20260926 20:49:19.561038 8201 raft_consensus.cc:2454] T 2aede59f334b4b8fa9bd69d4211c4d16 P 5ebea5fcb4064638863a64bdbf8bb189 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad2e8806cab4a71bcdc7e7372c36854 in term 0.
I20260926 20:49:19.561165 8272 leader_election.cc:302] T 2aede59f334b4b8fa9bd69d4211c4d16 P aad2e8806cab4a71bcdc7e7372c36854 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aad2e8806cab4a71bcdc7e7372c36854, b7762c6bd65f4238ac92779f523d6b1c; no voters:
I20260926 20:49:19.561293 8433 raft_consensus.cc:2781] T 2aede59f334b4b8fa9bd69d4211c4d16 P aad2e8806cab4a71bcdc7e7372c36854 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:49:19.561339 8433 raft_consensus.cc:482] T 2aede59f334b4b8fa9bd69d4211c4d16 P aad2e8806cab4a71bcdc7e7372c36854 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:49:19.561363 8433 raft_consensus.cc:3037] T 2aede59f334b4b8fa9bd69d4211c4d16 P aad2e8806cab4a71bcdc7e7372c36854 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:49:19.562077 8433 raft_consensus.cc:504] T 2aede59f334b4b8fa9bd69d4211c4d16 P aad2e8806cab4a71bcdc7e7372c36854 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } } peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } }
I20260926 20:49:19.562211 8433 leader_election.cc:288] T 2aede59f334b4b8fa9bd69d4211c4d16 P aad2e8806cab4a71bcdc7e7372c36854 [CANDIDATE]: Term 1 election: Requested vote from peers b7762c6bd65f4238ac92779f523d6b1c (127.0.98.1:38323), 5ebea5fcb4064638863a64bdbf8bb189 (127.0.98.2:43131)
I20260926 20:49:19.562407 8201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aede59f334b4b8fa9bd69d4211c4d16" candidate_uuid: "aad2e8806cab4a71bcdc7e7372c36854" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ebea5fcb4064638863a64bdbf8bb189"
I20260926 20:49:19.562409 8060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aede59f334b4b8fa9bd69d4211c4d16" candidate_uuid: "aad2e8806cab4a71bcdc7e7372c36854" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7762c6bd65f4238ac92779f523d6b1c"
I20260926 20:49:19.562491 8201 raft_consensus.cc:3037] T 2aede59f334b4b8fa9bd69d4211c4d16 P 5ebea5fcb4064638863a64bdbf8bb189 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:49:19.562492 8060 raft_consensus.cc:3037] T 2aede59f334b4b8fa9bd69d4211c4d16 P b7762c6bd65f4238ac92779f523d6b1c [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:49:19.563223 8060 raft_consensus.cc:2454] T 2aede59f334b4b8fa9bd69d4211c4d16 P b7762c6bd65f4238ac92779f523d6b1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad2e8806cab4a71bcdc7e7372c36854 in term 1.
I20260926 20:49:19.563223 8201 raft_consensus.cc:2454] T 2aede59f334b4b8fa9bd69d4211c4d16 P 5ebea5fcb4064638863a64bdbf8bb189 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad2e8806cab4a71bcdc7e7372c36854 in term 1.
I20260926 20:49:19.563392 8272 leader_election.cc:302] T 2aede59f334b4b8fa9bd69d4211c4d16 P aad2e8806cab4a71bcdc7e7372c36854 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aad2e8806cab4a71bcdc7e7372c36854, b7762c6bd65f4238ac92779f523d6b1c; no voters:
I20260926 20:49:19.563475 8433 raft_consensus.cc:2781] T 2aede59f334b4b8fa9bd69d4211c4d16 P aad2e8806cab4a71bcdc7e7372c36854 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:49:19.563532 8433 raft_consensus.cc:686] T 2aede59f334b4b8fa9bd69d4211c4d16 P aad2e8806cab4a71bcdc7e7372c36854 [term 1 LEADER]: Becoming Leader. State: Replica: aad2e8806cab4a71bcdc7e7372c36854, State: Running, Role: LEADER
I20260926 20:49:19.563587 8433 consensus_queue.cc:237] T 2aede59f334b4b8fa9bd69d4211c4d16 P aad2e8806cab4a71bcdc7e7372c36854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } } peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } }
I20260926 20:49:19.563905 8422 tablet_replica.cc:439] T 2aede59f334b4b8fa9bd69d4211c4d16 P aad2e8806cab4a71bcdc7e7372c36854: TxnStatusTablet state changed. Reason: New leader aad2e8806cab4a71bcdc7e7372c36854. Latest consensus state: current_term: 1 leader_uuid: "aad2e8806cab4a71bcdc7e7372c36854" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } } peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } } }
I20260926 20:49:19.563979 8422 tablet_replica.cc:442] T 2aede59f334b4b8fa9bd69d4211c4d16 P aad2e8806cab4a71bcdc7e7372c36854: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:49:19.564190 8457 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 20:49:19.564222 7904 catalog_manager.cc:5495] T 2aede59f334b4b8fa9bd69d4211c4d16 P aad2e8806cab4a71bcdc7e7372c36854 reported cstate change: term changed from 0 to 1, leader changed from <none> to aad2e8806cab4a71bcdc7e7372c36854 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "aad2e8806cab4a71bcdc7e7372c36854" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } health_report { overall_health: HEALTHY } } }
I20260926 20:49:19.564332 8457 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 20:49:19.599067 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:19.596393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33178 (local address 127.0.98.3:37051)
0926 20:49:19.596448 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:19.596450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:19.596467 (+ 17us) server_negotiation.cc:407] Connection header received
0926 20:49:19.596539 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:19.596540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:19.596550 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:19.596599 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:49:19.596743 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.597211 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.598324 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.598537 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.598597 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:19.598814 (+ 217us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:49:19.598847 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:49:19.598901 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:19.598986 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 20:49:19.599007 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 20:49:19.599123 8452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:19.596302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37051 (local address 127.0.0.1:33178)
0926 20:49:19.596395 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:19.596410 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:19.596463 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:19.596620 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:19.596621 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:19.596622 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:49:19.596678 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:19.596681 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.597288 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.597288 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:19.598225 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:19.598227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.598620 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.598621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:19.598687 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:19.598706 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:49:19.598922 (+ 216us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:49:19.598929 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 20:49:19.598984 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 20:49:19.599040 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 20:49:19.600554 8060 raft_consensus.cc:1261] T 2aede59f334b4b8fa9bd69d4211c4d16 P b7762c6bd65f4238ac92779f523d6b1c [term 1 FOLLOWER]: Refusing update from remote peer aad2e8806cab4a71bcdc7e7372c36854: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:49:19.600813 8201 raft_consensus.cc:1261] T 2aede59f334b4b8fa9bd69d4211c4d16 P 5ebea5fcb4064638863a64bdbf8bb189 [term 1 FOLLOWER]: Refusing update from remote peer aad2e8806cab4a71bcdc7e7372c36854: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:49:19.600797 8422 consensus_queue.cc:1035] T 2aede59f334b4b8fa9bd69d4211c4d16 P aad2e8806cab4a71bcdc7e7372c36854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:49:19.601081 8422 consensus_queue.cc:1035] T 2aede59f334b4b8fa9bd69d4211c4d16 P aad2e8806cab4a71bcdc7e7372c36854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:49:19.601431 8423 tablet_replica.cc:439] T 2aede59f334b4b8fa9bd69d4211c4d16 P b7762c6bd65f4238ac92779f523d6b1c: TxnStatusTablet state changed. Reason: New leader aad2e8806cab4a71bcdc7e7372c36854. Latest consensus state: current_term: 1 leader_uuid: "aad2e8806cab4a71bcdc7e7372c36854" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } } peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } } }
I20260926 20:49:19.601532 8423 tablet_replica.cc:442] T 2aede59f334b4b8fa9bd69d4211c4d16 P b7762c6bd65f4238ac92779f523d6b1c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:49:19.601924 8422 tablet_replica.cc:439] T 2aede59f334b4b8fa9bd69d4211c4d16 P aad2e8806cab4a71bcdc7e7372c36854: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aad2e8806cab4a71bcdc7e7372c36854" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } } peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } } }
I20260926 20:49:19.601999 8422 tablet_replica.cc:442] T 2aede59f334b4b8fa9bd69d4211c4d16 P aad2e8806cab4a71bcdc7e7372c36854: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:49:19.601951 8421 tablet_replica.cc:439] T 2aede59f334b4b8fa9bd69d4211c4d16 P 5ebea5fcb4064638863a64bdbf8bb189: TxnStatusTablet state changed. Reason: New leader aad2e8806cab4a71bcdc7e7372c36854. Latest consensus state: current_term: 1 leader_uuid: "aad2e8806cab4a71bcdc7e7372c36854" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } } peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } } }
I20260926 20:49:19.602027 8421 tablet_replica.cc:442] T 2aede59f334b4b8fa9bd69d4211c4d16 P 5ebea5fcb4064638863a64bdbf8bb189: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:49:19.602236 8428 mvcc.cc:205] Tried to move back new op lower bound from 7333706791321673728 to 7333706791172714496. Current Snapshot: MvccSnapshot[applied={T|T < 7333706791321673728 or (T in {7333706791321673728})}]
I20260926 20:49:19.602372 8430 mvcc.cc:205] Tried to move back new op lower bound from 7333706791321673728 to 7333706791172714496. Current Snapshot: MvccSnapshot[applied={T|T < 7333706791321673728 or (T in {7333706791321673728})}]
I20260926 20:49:19.602464 8423 tablet_replica.cc:439] T 2aede59f334b4b8fa9bd69d4211c4d16 P b7762c6bd65f4238ac92779f523d6b1c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aad2e8806cab4a71bcdc7e7372c36854" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } } peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } } }
I20260926 20:49:19.602536 8423 tablet_replica.cc:442] T 2aede59f334b4b8fa9bd69d4211c4d16 P b7762c6bd65f4238ac92779f523d6b1c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:49:19.603406 8433 tablet_replica.cc:439] T 2aede59f334b4b8fa9bd69d4211c4d16 P aad2e8806cab4a71bcdc7e7372c36854: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aad2e8806cab4a71bcdc7e7372c36854" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } } peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } } }
I20260926 20:49:19.603507 8433 tablet_replica.cc:442] T 2aede59f334b4b8fa9bd69d4211c4d16 P aad2e8806cab4a71bcdc7e7372c36854: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:49:19.603797 8421 tablet_replica.cc:439] T 2aede59f334b4b8fa9bd69d4211c4d16 P 5ebea5fcb4064638863a64bdbf8bb189: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aad2e8806cab4a71bcdc7e7372c36854" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } } peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } } }
I20260926 20:49:19.603878 8421 tablet_replica.cc:442] T 2aede59f334b4b8fa9bd69d4211c4d16 P 5ebea5fcb4064638863a64bdbf8bb189: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:49:19.607060 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:19.604458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48736 (local address 127.0.98.2:43131)
0926 20:49:19.604498 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:19.604500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:19.604508 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:49:19.604545 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:19.604546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:19.604565 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:19.604637 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:49:19.604765 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.605517 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.606257 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.606549 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.606618 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:19.606725 (+ 107us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:49:19.606852 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:49:19.606920 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:19.606941 (+ 21us) server_negotiation.cc:299] Negotiation successful
0926 20:49:19.606958 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 20:49:19.612038 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:19.609629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57633 (local address 127.0.98.3:37051)
0926 20:49:19.609685 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:19.609687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:19.609757 (+ 70us) server_negotiation.cc:407] Connection header received
0926 20:49:19.609830 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:19.609830 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:19.609840 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:19.609879 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:49:19.609991 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.610422 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.611405 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.611586 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.611783 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:19.611817 (+ 34us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:49:19.611858 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:49:19.611895 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:19.611973 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 20:49:19.611989 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 20:49:19.612102 8463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:19.609549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37051 (local address 127.0.98.2:57633)
0926 20:49:19.609702 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:19.609713 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:19.609772 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:19.609890 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:19.609891 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:19.609892 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:49:19.609940 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:19.609941 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.610487 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.610488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:19.611299 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:19.611301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.611662 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.611663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:19.611729 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:19.611760 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:49:19.611915 (+ 155us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:49:19.611917 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 20:49:19.611971 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 20:49:19.612019 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":84,"threads_started":1}
I20260926 20:49:19.614180 8060 raft_consensus.cc:1261] T 50dab45176a5469fbdc9452774754c9c P b7762c6bd65f4238ac92779f523d6b1c [term 1 FOLLOWER]: Refusing update from remote peer 5ebea5fcb4064638863a64bdbf8bb189: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:49:19.614300 8341 raft_consensus.cc:1261] T 50dab45176a5469fbdc9452774754c9c P aad2e8806cab4a71bcdc7e7372c36854 [term 1 FOLLOWER]: Refusing update from remote peer 5ebea5fcb4064638863a64bdbf8bb189: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:49:19.614405 8445 consensus_queue.cc:1035] T 50dab45176a5469fbdc9452774754c9c P 5ebea5fcb4064638863a64bdbf8bb189 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:49:19.614585 8445 consensus_queue.cc:1035] T 50dab45176a5469fbdc9452774754c9c P 5ebea5fcb4064638863a64bdbf8bb189 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:49:19.622566 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:19.619859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43131 (local address 127.0.98.3:43175)
0926 20:49:19.619977 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:19.619989 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:19.620036 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:19.620156 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:19.620157 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:19.620158 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:49:19.620192 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:19.620193 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.621006 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.621007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:19.621792 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:19.621795 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.622234 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.622234 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:19.622274 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:19.622286 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:49:19.622449 (+ 163us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:49:19.622451 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 20:49:19.622487 (+ 36us) client_negotiation.cc:240] Negotiation successful
0926 20:49:19.622519 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260926 20:49:19.622571 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:19.619917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43175 (local address 127.0.98.2:43131)
0926 20:49:19.619956 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:19.619958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:19.620022 (+ 64us) server_negotiation.cc:407] Connection header received
0926 20:49:19.620082 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:19.620082 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:19.620093 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:19.620156 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:49:19.620227 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.620911 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.621868 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.622162 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.622230 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:19.622360 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:49:19.622392 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:49:19.622443 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:19.622490 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 20:49:19.622507 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 20:49:19.631696 7903 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:49:19.631865 7903 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:49:19.632493 7903 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b012d088b8243c290a80ca96476dd0c: Sending DeleteTablet for 3 replicas of tablet 50dab45176a5469fbdc9452774754c9c
I20260926 20:49:19.632727 8041 tablet_service.cc:1514] Processing DeleteTablet for tablet 50dab45176a5469fbdc9452774754c9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:49:19 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53438
I20260926 20:49:19.632834 8321 tablet_service.cc:1514] Processing DeleteTablet for tablet 50dab45176a5469fbdc9452774754c9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:49:19 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33166
I20260926 20:49:19.632834 8440 tablet_replica.cc:330] T 50dab45176a5469fbdc9452774754c9c P b7762c6bd65f4238ac92779f523d6b1c: stopping tablet replica
I20260926 20:49:19.632903 8440 raft_consensus.cc:2229] T 50dab45176a5469fbdc9452774754c9c P b7762c6bd65f4238ac92779f523d6b1c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:49:19.632942 8440 raft_consensus.cc:2258] T 50dab45176a5469fbdc9452774754c9c P b7762c6bd65f4238ac92779f523d6b1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:49:19.632948 8439 tablet_replica.cc:330] T 50dab45176a5469fbdc9452774754c9c P aad2e8806cab4a71bcdc7e7372c36854: stopping tablet replica
I20260926 20:49:19.632998 8439 raft_consensus.cc:2229] T 50dab45176a5469fbdc9452774754c9c P aad2e8806cab4a71bcdc7e7372c36854 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:49:19.633028 8439 raft_consensus.cc:2258] T 50dab45176a5469fbdc9452774754c9c P aad2e8806cab4a71bcdc7e7372c36854 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:49:19.633282 8440 ts_tablet_manager.cc:1905] T 50dab45176a5469fbdc9452774754c9c P b7762c6bd65f4238ac92779f523d6b1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:49:19.633332 8439 ts_tablet_manager.cc:1905] T 50dab45176a5469fbdc9452774754c9c P aad2e8806cab4a71bcdc7e7372c36854: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:49:19.635267 8440 ts_tablet_manager.cc:1918] T 50dab45176a5469fbdc9452774754c9c P b7762c6bd65f4238ac92779f523d6b1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:49:19.635322 8440 log.cc:1196] T 50dab45176a5469fbdc9452774754c9c P b7762c6bd65f4238ac92779f523d6b1c: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-0/wal/wals/50dab45176a5469fbdc9452774754c9c
I20260926 20:49:19.635540 8440 ts_tablet_manager.cc:1939] T 50dab45176a5469fbdc9452774754c9c P b7762c6bd65f4238ac92779f523d6b1c: Deleting consensus metadata
I20260926 20:49:19.635878 8181 tablet_service.cc:1514] Processing DeleteTablet for tablet 50dab45176a5469fbdc9452774754c9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:49:19 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48720
I20260926 20:49:19.635895 7888 catalog_manager.cc:4841] TS b7762c6bd65f4238ac92779f523d6b1c (127.0.98.1:38323): tablet 50dab45176a5469fbdc9452774754c9c (table test-table [id=7da1dec31b91453f8466227f3718565a]) successfully deleted
I20260926 20:49:19.635987 8438 tablet_replica.cc:330] T 50dab45176a5469fbdc9452774754c9c P 5ebea5fcb4064638863a64bdbf8bb189: stopping tablet replica
I20260926 20:49:19.636036 8438 raft_consensus.cc:2229] T 50dab45176a5469fbdc9452774754c9c P 5ebea5fcb4064638863a64bdbf8bb189 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:49:19.636133 8438 raft_consensus.cc:2258] T 50dab45176a5469fbdc9452774754c9c P 5ebea5fcb4064638863a64bdbf8bb189 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:49:19.636267 8439 ts_tablet_manager.cc:1918] T 50dab45176a5469fbdc9452774754c9c P aad2e8806cab4a71bcdc7e7372c36854: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:49:19.636315 8439 log.cc:1196] T 50dab45176a5469fbdc9452774754c9c P aad2e8806cab4a71bcdc7e7372c36854: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-2/wal/wals/50dab45176a5469fbdc9452774754c9c
I20260926 20:49:19.636471 8438 ts_tablet_manager.cc:1905] T 50dab45176a5469fbdc9452774754c9c P 5ebea5fcb4064638863a64bdbf8bb189: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:49:19.636530 8439 ts_tablet_manager.cc:1939] T 50dab45176a5469fbdc9452774754c9c P aad2e8806cab4a71bcdc7e7372c36854: Deleting consensus metadata
I20260926 20:49:19.636823 7889 catalog_manager.cc:4841] TS aad2e8806cab4a71bcdc7e7372c36854 (127.0.98.3:37051): tablet 50dab45176a5469fbdc9452774754c9c (table test-table [id=7da1dec31b91453f8466227f3718565a]) successfully deleted
I20260926 20:49:19.638005 8438 ts_tablet_manager.cc:1918] T 50dab45176a5469fbdc9452774754c9c P 5ebea5fcb4064638863a64bdbf8bb189: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:49:19.638072 8438 log.cc:1196] T 50dab45176a5469fbdc9452774754c9c P 5ebea5fcb4064638863a64bdbf8bb189: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-1/wal/wals/50dab45176a5469fbdc9452774754c9c
I20260926 20:49:19.638276 8438 ts_tablet_manager.cc:1939] T 50dab45176a5469fbdc9452774754c9c P 5ebea5fcb4064638863a64bdbf8bb189: Deleting consensus metadata
I20260926 20:49:19.638628 7888 catalog_manager.cc:4841] TS 5ebea5fcb4064638863a64bdbf8bb189 (127.0.98.2:43131): tablet 50dab45176a5469fbdc9452774754c9c (table test-table [id=7da1dec31b91453f8466227f3718565a]) successfully deleted
Sep 26 20:49:19 dist-test-slave-hwlj krb5kdc[7859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 20:49:19 dist-test-slave-hwlj krb5kdc[7859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:49:19 dist-test-slave-hwlj krb5kdc[7859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 20:49:19.659092 7972 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:19.653536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45894 (local address 127.0.98.62:32897)
0926 20:49:19.653591 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:19.653593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:19.653754 (+ 161us) server_negotiation.cc:407] Connection header received
0926 20:49:19.653835 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:19.653836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:19.653848 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:19.653898 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:19.654523 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.655173 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.656056 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.656259 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.656331 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:19.659044 (+ 2713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45894: BlockingRecv error: recv got EOF from 127.0.0.1:45894 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 26 20:49:19 dist-test-slave-hwlj krb5kdc[7859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 20:49:19.667176 7972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:19.660182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45908 (local address 127.0.98.62:32897)
0926 20:49:19.660217 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:19.660219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:19.660349 (+ 130us) server_negotiation.cc:407] Connection header received
0926 20:49:19.660418 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:19.660419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:19.660428 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:19.660498 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:19.660954 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.661421 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.662094 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.662279 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.662368 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:19.664628 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:19.664629 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:19.664630 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:19.664637 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:19.666061 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:19.666417 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:19.666418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:19.666418 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:19.666478 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:19.666680 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:19.666681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:19.666682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:19.666794 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:19.666894 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:19.666992 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 20:49:19.667066 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 20:49:19.668737 7903 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45908:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:49:19.668922 7903 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:49:19.670223 8321 tablet_service.cc:1467] Processing CreateTablet for tablet cff01f910cde482b999584d7d01368e2 (DEFAULT_TABLE table=test-table [id=c844b58909c44b3e90c4189258d88898]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:49:19.670326 8181 tablet_service.cc:1467] Processing CreateTablet for tablet cff01f910cde482b999584d7d01368e2 (DEFAULT_TABLE table=test-table [id=c844b58909c44b3e90c4189258d88898]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:49:19.670360 8321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cff01f910cde482b999584d7d01368e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:49:19.670428 8181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cff01f910cde482b999584d7d01368e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:49:19.671623 8415 tablet_bootstrap.cc:492] T cff01f910cde482b999584d7d01368e2 P 5ebea5fcb4064638863a64bdbf8bb189: Bootstrap starting.
I20260926 20:49:19.671663 8041 tablet_service.cc:1467] Processing CreateTablet for tablet cff01f910cde482b999584d7d01368e2 (DEFAULT_TABLE table=test-table [id=c844b58909c44b3e90c4189258d88898]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:49:19.671805 8041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cff01f910cde482b999584d7d01368e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:49:19.672078 8415 tablet_bootstrap.cc:654] T cff01f910cde482b999584d7d01368e2 P 5ebea5fcb4064638863a64bdbf8bb189: Neither blocks nor log segments found. Creating new log.
I20260926 20:49:19.672724 8415 tablet_bootstrap.cc:492] T cff01f910cde482b999584d7d01368e2 P 5ebea5fcb4064638863a64bdbf8bb189: No bootstrap required, opened a new log
I20260926 20:49:19.672765 8415 ts_tablet_manager.cc:1397] T cff01f910cde482b999584d7d01368e2 P 5ebea5fcb4064638863a64bdbf8bb189: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:49:19.672875 8416 tablet_bootstrap.cc:492] T cff01f910cde482b999584d7d01368e2 P b7762c6bd65f4238ac92779f523d6b1c: Bootstrap starting.
I20260926 20:49:19.672914 8418 tablet_bootstrap.cc:492] T cff01f910cde482b999584d7d01368e2 P aad2e8806cab4a71bcdc7e7372c36854: Bootstrap starting.
I20260926 20:49:19.672900 8415 raft_consensus.cc:348] T cff01f910cde482b999584d7d01368e2 P 5ebea5fcb4064638863a64bdbf8bb189 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } } peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } }
I20260926 20:49:19.672962 8415 raft_consensus.cc:374] T cff01f910cde482b999584d7d01368e2 P 5ebea5fcb4064638863a64bdbf8bb189 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:49:19.672983 8415 raft_consensus.cc:729] T cff01f910cde482b999584d7d01368e2 P 5ebea5fcb4064638863a64bdbf8bb189 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ebea5fcb4064638863a64bdbf8bb189, State: Initialized, Role: FOLLOWER
I20260926 20:49:19.673027 8415 consensus_queue.cc:260] T cff01f910cde482b999584d7d01368e2 P 5ebea5fcb4064638863a64bdbf8bb189 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } } peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } }
I20260926 20:49:19.673121 8415 ts_tablet_manager.cc:1428] T cff01f910cde482b999584d7d01368e2 P 5ebea5fcb4064638863a64bdbf8bb189: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:19.673211 8418 tablet_bootstrap.cc:654] T cff01f910cde482b999584d7d01368e2 P aad2e8806cab4a71bcdc7e7372c36854: Neither blocks nor log segments found. Creating new log.
I20260926 20:49:19.673275 8416 tablet_bootstrap.cc:654] T cff01f910cde482b999584d7d01368e2 P b7762c6bd65f4238ac92779f523d6b1c: Neither blocks nor log segments found. Creating new log.
I20260926 20:49:19.674115 8416 tablet_bootstrap.cc:492] T cff01f910cde482b999584d7d01368e2 P b7762c6bd65f4238ac92779f523d6b1c: No bootstrap required, opened a new log
I20260926 20:49:19.674162 8416 ts_tablet_manager.cc:1397] T cff01f910cde482b999584d7d01368e2 P b7762c6bd65f4238ac92779f523d6b1c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 20:49:19.674304 8416 raft_consensus.cc:348] T cff01f910cde482b999584d7d01368e2 P b7762c6bd65f4238ac92779f523d6b1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } } peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } }
I20260926 20:49:19.674386 8416 raft_consensus.cc:374] T cff01f910cde482b999584d7d01368e2 P b7762c6bd65f4238ac92779f523d6b1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:49:19.674410 8416 raft_consensus.cc:729] T cff01f910cde482b999584d7d01368e2 P b7762c6bd65f4238ac92779f523d6b1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7762c6bd65f4238ac92779f523d6b1c, State: Initialized, Role: FOLLOWER
I20260926 20:49:19.674454 8416 consensus_queue.cc:260] T cff01f910cde482b999584d7d01368e2 P b7762c6bd65f4238ac92779f523d6b1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } } peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } }
I20260926 20:49:19.674575 8416 ts_tablet_manager.cc:1428] T cff01f910cde482b999584d7d01368e2 P b7762c6bd65f4238ac92779f523d6b1c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:19.675132 8418 tablet_bootstrap.cc:492] T cff01f910cde482b999584d7d01368e2 P aad2e8806cab4a71bcdc7e7372c36854: No bootstrap required, opened a new log
I20260926 20:49:19.675181 8418 ts_tablet_manager.cc:1397] T cff01f910cde482b999584d7d01368e2 P aad2e8806cab4a71bcdc7e7372c36854: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:49:19.675287 8418 raft_consensus.cc:348] T cff01f910cde482b999584d7d01368e2 P aad2e8806cab4a71bcdc7e7372c36854 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } } peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } }
I20260926 20:49:19.675350 8418 raft_consensus.cc:374] T cff01f910cde482b999584d7d01368e2 P aad2e8806cab4a71bcdc7e7372c36854 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:49:19.675364 8418 raft_consensus.cc:729] T cff01f910cde482b999584d7d01368e2 P aad2e8806cab4a71bcdc7e7372c36854 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad2e8806cab4a71bcdc7e7372c36854, State: Initialized, Role: FOLLOWER
I20260926 20:49:19.675419 8418 consensus_queue.cc:260] T cff01f910cde482b999584d7d01368e2 P aad2e8806cab4a71bcdc7e7372c36854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } } peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } }
I20260926 20:49:19.675508 8418 ts_tablet_manager.cc:1428] T cff01f910cde482b999584d7d01368e2 P aad2e8806cab4a71bcdc7e7372c36854: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:19.722472 8421 raft_consensus.cc:482] T cff01f910cde482b999584d7d01368e2 P 5ebea5fcb4064638863a64bdbf8bb189 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:49:19.722594 8421 raft_consensus.cc:504] T cff01f910cde482b999584d7d01368e2 P 5ebea5fcb4064638863a64bdbf8bb189 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } } peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } }
I20260926 20:49:19.722767 8421 leader_election.cc:288] T cff01f910cde482b999584d7d01368e2 P 5ebea5fcb4064638863a64bdbf8bb189 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7762c6bd65f4238ac92779f523d6b1c (127.0.98.1:38323), aad2e8806cab4a71bcdc7e7372c36854 (127.0.98.3:37051)
I20260926 20:49:19.723043 8061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cff01f910cde482b999584d7d01368e2" candidate_uuid: "5ebea5fcb4064638863a64bdbf8bb189" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7762c6bd65f4238ac92779f523d6b1c" is_pre_election: true
I20260926 20:49:19.723043 8341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cff01f910cde482b999584d7d01368e2" candidate_uuid: "5ebea5fcb4064638863a64bdbf8bb189" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad2e8806cab4a71bcdc7e7372c36854" is_pre_election: true
I20260926 20:49:19.723137 8061 raft_consensus.cc:2454] T cff01f910cde482b999584d7d01368e2 P b7762c6bd65f4238ac92779f523d6b1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ebea5fcb4064638863a64bdbf8bb189 in term 0.
I20260926 20:49:19.723137 8341 raft_consensus.cc:2454] T cff01f910cde482b999584d7d01368e2 P aad2e8806cab4a71bcdc7e7372c36854 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ebea5fcb4064638863a64bdbf8bb189 in term 0.
I20260926 20:49:19.723302 8133 leader_election.cc:302] T cff01f910cde482b999584d7d01368e2 P 5ebea5fcb4064638863a64bdbf8bb189 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ebea5fcb4064638863a64bdbf8bb189, aad2e8806cab4a71bcdc7e7372c36854; no voters:
I20260926 20:49:19.723495 8421 raft_consensus.cc:2781] T cff01f910cde482b999584d7d01368e2 P 5ebea5fcb4064638863a64bdbf8bb189 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:49:19.723539 8421 raft_consensus.cc:482] T cff01f910cde482b999584d7d01368e2 P 5ebea5fcb4064638863a64bdbf8bb189 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:49:19.723567 8421 raft_consensus.cc:3037] T cff01f910cde482b999584d7d01368e2 P 5ebea5fcb4064638863a64bdbf8bb189 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:49:19.724375 8421 raft_consensus.cc:504] T cff01f910cde482b999584d7d01368e2 P 5ebea5fcb4064638863a64bdbf8bb189 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } } peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } }
I20260926 20:49:19.724532 8421 leader_election.cc:288] T cff01f910cde482b999584d7d01368e2 P 5ebea5fcb4064638863a64bdbf8bb189 [CANDIDATE]: Term 1 election: Requested vote from peers b7762c6bd65f4238ac92779f523d6b1c (127.0.98.1:38323), aad2e8806cab4a71bcdc7e7372c36854 (127.0.98.3:37051)
I20260926 20:49:19.724740 8341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cff01f910cde482b999584d7d01368e2" candidate_uuid: "5ebea5fcb4064638863a64bdbf8bb189" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad2e8806cab4a71bcdc7e7372c36854"
I20260926 20:49:19.724741 8061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cff01f910cde482b999584d7d01368e2" candidate_uuid: "5ebea5fcb4064638863a64bdbf8bb189" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7762c6bd65f4238ac92779f523d6b1c"
I20260926 20:49:19.724808 8061 raft_consensus.cc:3037] T cff01f910cde482b999584d7d01368e2 P b7762c6bd65f4238ac92779f523d6b1c [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:49:19.724808 8341 raft_consensus.cc:3037] T cff01f910cde482b999584d7d01368e2 P aad2e8806cab4a71bcdc7e7372c36854 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:49:19.725697 8061 raft_consensus.cc:2454] T cff01f910cde482b999584d7d01368e2 P b7762c6bd65f4238ac92779f523d6b1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ebea5fcb4064638863a64bdbf8bb189 in term 1.
I20260926 20:49:19.725768 8341 raft_consensus.cc:2454] T cff01f910cde482b999584d7d01368e2 P aad2e8806cab4a71bcdc7e7372c36854 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ebea5fcb4064638863a64bdbf8bb189 in term 1.
I20260926 20:49:19.725869 8132 leader_election.cc:302] T cff01f910cde482b999584d7d01368e2 P 5ebea5fcb4064638863a64bdbf8bb189 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ebea5fcb4064638863a64bdbf8bb189, b7762c6bd65f4238ac92779f523d6b1c; no voters:
I20260926 20:49:19.725957 8421 raft_consensus.cc:2781] T cff01f910cde482b999584d7d01368e2 P 5ebea5fcb4064638863a64bdbf8bb189 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:49:19.725994 8421 raft_consensus.cc:686] T cff01f910cde482b999584d7d01368e2 P 5ebea5fcb4064638863a64bdbf8bb189 [term 1 LEADER]: Becoming Leader. State: Replica: 5ebea5fcb4064638863a64bdbf8bb189, State: Running, Role: LEADER
I20260926 20:49:19.726032 8421 consensus_queue.cc:237] T cff01f910cde482b999584d7d01368e2 P 5ebea5fcb4064638863a64bdbf8bb189 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } } peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } }
I20260926 20:49:19.726665 7903 catalog_manager.cc:5495] T cff01f910cde482b999584d7d01368e2 P 5ebea5fcb4064638863a64bdbf8bb189 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ebea5fcb4064638863a64bdbf8bb189 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "5ebea5fcb4064638863a64bdbf8bb189" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } health_report { overall_health: UNKNOWN } } }
I20260926 20:49:19.735039 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:19.732485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48744 (local address 127.0.98.2:43131)
0926 20:49:19.732537 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:19.732539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:19.732548 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:49:19.732575 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:19.732576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:19.732589 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:19.732655 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:49:19.732785 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.733507 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.734265 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.734538 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.734617 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:19.734774 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:49:19.734808 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:49:19.734864 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:19.734950 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 20:49:19.734969 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 20:49:19.735899 8341 raft_consensus.cc:1261] T cff01f910cde482b999584d7d01368e2 P aad2e8806cab4a71bcdc7e7372c36854 [term 1 FOLLOWER]: Refusing update from remote peer 5ebea5fcb4064638863a64bdbf8bb189: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:49:19.736091 8061 raft_consensus.cc:1261] T cff01f910cde482b999584d7d01368e2 P b7762c6bd65f4238ac92779f523d6b1c [term 1 FOLLOWER]: Refusing update from remote peer 5ebea5fcb4064638863a64bdbf8bb189: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:49:19.736168 8466 consensus_queue.cc:1035] T cff01f910cde482b999584d7d01368e2 P 5ebea5fcb4064638863a64bdbf8bb189 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:49:19.736478 8466 consensus_queue.cc:1035] T cff01f910cde482b999584d7d01368e2 P 5ebea5fcb4064638863a64bdbf8bb189 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:49:19.739312 7902 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:49:19.739418 7902 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:49:19.740068 7902 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b012d088b8243c290a80ca96476dd0c: Sending DeleteTablet for 3 replicas of tablet cff01f910cde482b999584d7d01368e2
I20260926 20:49:19.740305 8041 tablet_service.cc:1514] Processing DeleteTablet for tablet cff01f910cde482b999584d7d01368e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:49:19 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53438
I20260926 20:49:19.740415 8440 tablet_replica.cc:330] T cff01f910cde482b999584d7d01368e2 P b7762c6bd65f4238ac92779f523d6b1c: stopping tablet replica
I20260926 20:49:19.740412 8181 tablet_service.cc:1514] Processing DeleteTablet for tablet cff01f910cde482b999584d7d01368e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:49:19 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48720
I20260926 20:49:19.740468 8440 raft_consensus.cc:2229] T cff01f910cde482b999584d7d01368e2 P b7762c6bd65f4238ac92779f523d6b1c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:49:19.740487 8438 tablet_replica.cc:330] T cff01f910cde482b999584d7d01368e2 P 5ebea5fcb4064638863a64bdbf8bb189: stopping tablet replica
I20260926 20:49:19.740515 8440 raft_consensus.cc:2258] T cff01f910cde482b999584d7d01368e2 P b7762c6bd65f4238ac92779f523d6b1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:49:19.740526 8438 raft_consensus.cc:2229] T cff01f910cde482b999584d7d01368e2 P 5ebea5fcb4064638863a64bdbf8bb189 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:49:19.740581 8438 raft_consensus.cc:2258] T cff01f910cde482b999584d7d01368e2 P 5ebea5fcb4064638863a64bdbf8bb189 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:49:19.740917 8321 tablet_service.cc:1514] Processing DeleteTablet for tablet cff01f910cde482b999584d7d01368e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:49:19 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33166
I20260926 20:49:19.741004 8439 tablet_replica.cc:330] T cff01f910cde482b999584d7d01368e2 P aad2e8806cab4a71bcdc7e7372c36854: stopping tablet replica
I20260926 20:49:19.740990 8438 ts_tablet_manager.cc:1905] T cff01f910cde482b999584d7d01368e2 P 5ebea5fcb4064638863a64bdbf8bb189: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:49:19.740933 7902 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45908:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 20:49:19.741045 8439 raft_consensus.cc:2229] T cff01f910cde482b999584d7d01368e2 P aad2e8806cab4a71bcdc7e7372c36854 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260926 20:49:19.741072 7902 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:49:19.741081 8439 raft_consensus.cc:2258] T cff01f910cde482b999584d7d01368e2 P aad2e8806cab4a71bcdc7e7372c36854 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:49:19.741389 8439 ts_tablet_manager.cc:1905] T cff01f910cde482b999584d7d01368e2 P aad2e8806cab4a71bcdc7e7372c36854: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:49:19.742419 8321 tablet_service.cc:1467] Processing CreateTablet for tablet 4c7a47aa06c34037be1dc4d42ee6c69e (DEFAULT_TABLE table=test-table [id=6053e0d692bb46b8b84eb01b1bd5a821]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:49:19.742513 8321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c7a47aa06c34037be1dc4d42ee6c69e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:49:19.742686 8438 ts_tablet_manager.cc:1918] T cff01f910cde482b999584d7d01368e2 P 5ebea5fcb4064638863a64bdbf8bb189: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:49:19.742751 8438 log.cc:1196] T cff01f910cde482b999584d7d01368e2 P 5ebea5fcb4064638863a64bdbf8bb189: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-1/wal/wals/cff01f910cde482b999584d7d01368e2
I20260926 20:49:19.742889 8440 ts_tablet_manager.cc:1905] T cff01f910cde482b999584d7d01368e2 P b7762c6bd65f4238ac92779f523d6b1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:49:19.742961 8439 ts_tablet_manager.cc:1918] T cff01f910cde482b999584d7d01368e2 P aad2e8806cab4a71bcdc7e7372c36854: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:49:19.743009 8439 log.cc:1196] T cff01f910cde482b999584d7d01368e2 P aad2e8806cab4a71bcdc7e7372c36854: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-2/wal/wals/cff01f910cde482b999584d7d01368e2
I20260926 20:49:19.743009 8438 ts_tablet_manager.cc:1939] T cff01f910cde482b999584d7d01368e2 P 5ebea5fcb4064638863a64bdbf8bb189: Deleting consensus metadata
I20260926 20:49:19.743217 8439 ts_tablet_manager.cc:1939] T cff01f910cde482b999584d7d01368e2 P aad2e8806cab4a71bcdc7e7372c36854: Deleting consensus metadata
I20260926 20:49:19.743706 8418 tablet_bootstrap.cc:492] T 4c7a47aa06c34037be1dc4d42ee6c69e P aad2e8806cab4a71bcdc7e7372c36854: Bootstrap starting.
I20260926 20:49:19.744028 8418 tablet_bootstrap.cc:654] T 4c7a47aa06c34037be1dc4d42ee6c69e P aad2e8806cab4a71bcdc7e7372c36854: Neither blocks nor log segments found. Creating new log.
I20260926 20:49:19.744416 8440 ts_tablet_manager.cc:1918] T cff01f910cde482b999584d7d01368e2 P b7762c6bd65f4238ac92779f523d6b1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:49:19.744481 8440 log.cc:1196] T cff01f910cde482b999584d7d01368e2 P b7762c6bd65f4238ac92779f523d6b1c: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-0/wal/wals/cff01f910cde482b999584d7d01368e2
I20260926 20:49:19.744526 8418 tablet_bootstrap.cc:492] T 4c7a47aa06c34037be1dc4d42ee6c69e P aad2e8806cab4a71bcdc7e7372c36854: No bootstrap required, opened a new log
I20260926 20:49:19.744544 7888 catalog_manager.cc:4841] TS 5ebea5fcb4064638863a64bdbf8bb189 (127.0.98.2:43131): tablet cff01f910cde482b999584d7d01368e2 (table test-table [id=c844b58909c44b3e90c4189258d88898]) successfully deleted
I20260926 20:49:19.744567 8418 ts_tablet_manager.cc:1397] T 4c7a47aa06c34037be1dc4d42ee6c69e P aad2e8806cab4a71bcdc7e7372c36854: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:49:19.744688 8440 ts_tablet_manager.cc:1939] T cff01f910cde482b999584d7d01368e2 P b7762c6bd65f4238ac92779f523d6b1c: Deleting consensus metadata
I20260926 20:49:19.744671 8418 raft_consensus.cc:348] T 4c7a47aa06c34037be1dc4d42ee6c69e P aad2e8806cab4a71bcdc7e7372c36854 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } } peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } }
I20260926 20:49:19.744736 8418 raft_consensus.cc:374] T 4c7a47aa06c34037be1dc4d42ee6c69e P aad2e8806cab4a71bcdc7e7372c36854 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:49:19.744728 8181 tablet_service.cc:1467] Processing CreateTablet for tablet 4c7a47aa06c34037be1dc4d42ee6c69e (DEFAULT_TABLE table=test-table [id=6053e0d692bb46b8b84eb01b1bd5a821]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:49:19.744756 8418 raft_consensus.cc:729] T 4c7a47aa06c34037be1dc4d42ee6c69e P aad2e8806cab4a71bcdc7e7372c36854 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad2e8806cab4a71bcdc7e7372c36854, State: Initialized, Role: FOLLOWER
I20260926 20:49:19.744820 8181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c7a47aa06c34037be1dc4d42ee6c69e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:49:19.744787 8418 consensus_queue.cc:260] T 4c7a47aa06c34037be1dc4d42ee6c69e P aad2e8806cab4a71bcdc7e7372c36854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } } peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } }
I20260926 20:49:19.744896 8418 ts_tablet_manager.cc:1428] T 4c7a47aa06c34037be1dc4d42ee6c69e P aad2e8806cab4a71bcdc7e7372c36854: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:19.745790 8415 tablet_bootstrap.cc:492] T 4c7a47aa06c34037be1dc4d42ee6c69e P 5ebea5fcb4064638863a64bdbf8bb189: Bootstrap starting.
I20260926 20:49:19.745901 7888 catalog_manager.cc:4841] TS b7762c6bd65f4238ac92779f523d6b1c (127.0.98.1:38323): tablet cff01f910cde482b999584d7d01368e2 (table test-table [id=c844b58909c44b3e90c4189258d88898]) successfully deleted
I20260926 20:49:19.745958 8041 tablet_service.cc:1467] Processing CreateTablet for tablet 4c7a47aa06c34037be1dc4d42ee6c69e (DEFAULT_TABLE table=test-table [id=6053e0d692bb46b8b84eb01b1bd5a821]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:49:19.746047 8041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c7a47aa06c34037be1dc4d42ee6c69e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:49:19.746217 8415 tablet_bootstrap.cc:654] T 4c7a47aa06c34037be1dc4d42ee6c69e P 5ebea5fcb4064638863a64bdbf8bb189: Neither blocks nor log segments found. Creating new log.
I20260926 20:49:19.746893 8415 tablet_bootstrap.cc:492] T 4c7a47aa06c34037be1dc4d42ee6c69e P 5ebea5fcb4064638863a64bdbf8bb189: No bootstrap required, opened a new log
I20260926 20:49:19.746956 8415 ts_tablet_manager.cc:1397] T 4c7a47aa06c34037be1dc4d42ee6c69e P 5ebea5fcb4064638863a64bdbf8bb189: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:49:19.747072 8416 tablet_bootstrap.cc:492] T 4c7a47aa06c34037be1dc4d42ee6c69e P b7762c6bd65f4238ac92779f523d6b1c: Bootstrap starting.
I20260926 20:49:19.747099 8415 raft_consensus.cc:348] T 4c7a47aa06c34037be1dc4d42ee6c69e P 5ebea5fcb4064638863a64bdbf8bb189 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } } peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } }
I20260926 20:49:19.747167 8415 raft_consensus.cc:374] T 4c7a47aa06c34037be1dc4d42ee6c69e P 5ebea5fcb4064638863a64bdbf8bb189 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:49:19.747188 8415 raft_consensus.cc:729] T 4c7a47aa06c34037be1dc4d42ee6c69e P 5ebea5fcb4064638863a64bdbf8bb189 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ebea5fcb4064638863a64bdbf8bb189, State: Initialized, Role: FOLLOWER
I20260926 20:49:19.747231 8415 consensus_queue.cc:260] T 4c7a47aa06c34037be1dc4d42ee6c69e P 5ebea5fcb4064638863a64bdbf8bb189 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } } peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } }
I20260926 20:49:19.747316 8415 ts_tablet_manager.cc:1428] T 4c7a47aa06c34037be1dc4d42ee6c69e P 5ebea5fcb4064638863a64bdbf8bb189: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 20:49:19.747396 7889 catalog_manager.cc:4841] TS aad2e8806cab4a71bcdc7e7372c36854 (127.0.98.3:37051): tablet cff01f910cde482b999584d7d01368e2 (table test-table [id=c844b58909c44b3e90c4189258d88898]) successfully deleted
I20260926 20:49:19.747699 8416 tablet_bootstrap.cc:654] T 4c7a47aa06c34037be1dc4d42ee6c69e P b7762c6bd65f4238ac92779f523d6b1c: Neither blocks nor log segments found. Creating new log.
I20260926 20:49:19.748389 8416 tablet_bootstrap.cc:492] T 4c7a47aa06c34037be1dc4d42ee6c69e P b7762c6bd65f4238ac92779f523d6b1c: No bootstrap required, opened a new log
I20260926 20:49:19.748445 8416 ts_tablet_manager.cc:1397] T 4c7a47aa06c34037be1dc4d42ee6c69e P b7762c6bd65f4238ac92779f523d6b1c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 20:49:19.748586 8416 raft_consensus.cc:348] T 4c7a47aa06c34037be1dc4d42ee6c69e P b7762c6bd65f4238ac92779f523d6b1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } } peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } }
I20260926 20:49:19.748656 8416 raft_consensus.cc:374] T 4c7a47aa06c34037be1dc4d42ee6c69e P b7762c6bd65f4238ac92779f523d6b1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:49:19.748697 8416 raft_consensus.cc:729] T 4c7a47aa06c34037be1dc4d42ee6c69e P b7762c6bd65f4238ac92779f523d6b1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7762c6bd65f4238ac92779f523d6b1c, State: Initialized, Role: FOLLOWER
I20260926 20:49:19.748745 8416 consensus_queue.cc:260] T 4c7a47aa06c34037be1dc4d42ee6c69e P b7762c6bd65f4238ac92779f523d6b1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } } peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } }
I20260926 20:49:19.748840 8416 ts_tablet_manager.cc:1428] T 4c7a47aa06c34037be1dc4d42ee6c69e P b7762c6bd65f4238ac92779f523d6b1c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:19.761932 8433 raft_consensus.cc:482] T 4c7a47aa06c34037be1dc4d42ee6c69e P aad2e8806cab4a71bcdc7e7372c36854 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:49:19.762058 8433 raft_consensus.cc:504] T 4c7a47aa06c34037be1dc4d42ee6c69e P aad2e8806cab4a71bcdc7e7372c36854 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } } peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } }
I20260926 20:49:19.762234 8433 leader_election.cc:288] T 4c7a47aa06c34037be1dc4d42ee6c69e P aad2e8806cab4a71bcdc7e7372c36854 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7762c6bd65f4238ac92779f523d6b1c (127.0.98.1:38323), 5ebea5fcb4064638863a64bdbf8bb189 (127.0.98.2:43131)
I20260926 20:49:19.762434 8061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7a47aa06c34037be1dc4d42ee6c69e" candidate_uuid: "aad2e8806cab4a71bcdc7e7372c36854" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7762c6bd65f4238ac92779f523d6b1c" is_pre_election: true
I20260926 20:49:19.762444 8201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7a47aa06c34037be1dc4d42ee6c69e" candidate_uuid: "aad2e8806cab4a71bcdc7e7372c36854" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ebea5fcb4064638863a64bdbf8bb189" is_pre_election: true
I20260926 20:49:19.762547 8201 raft_consensus.cc:2454] T 4c7a47aa06c34037be1dc4d42ee6c69e P 5ebea5fcb4064638863a64bdbf8bb189 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad2e8806cab4a71bcdc7e7372c36854 in term 0.
I20260926 20:49:19.762542 8061 raft_consensus.cc:2454] T 4c7a47aa06c34037be1dc4d42ee6c69e P b7762c6bd65f4238ac92779f523d6b1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad2e8806cab4a71bcdc7e7372c36854 in term 0.
I20260926 20:49:19.762692 8272 leader_election.cc:302] T 4c7a47aa06c34037be1dc4d42ee6c69e P aad2e8806cab4a71bcdc7e7372c36854 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ebea5fcb4064638863a64bdbf8bb189, aad2e8806cab4a71bcdc7e7372c36854; no voters:
I20260926 20:49:19.762784 8433 raft_consensus.cc:2781] T 4c7a47aa06c34037be1dc4d42ee6c69e P aad2e8806cab4a71bcdc7e7372c36854 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:49:19.762854 8433 raft_consensus.cc:482] T 4c7a47aa06c34037be1dc4d42ee6c69e P aad2e8806cab4a71bcdc7e7372c36854 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:49:19.762885 8433 raft_consensus.cc:3037] T 4c7a47aa06c34037be1dc4d42ee6c69e P aad2e8806cab4a71bcdc7e7372c36854 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:49:19.763566 8433 raft_consensus.cc:504] T 4c7a47aa06c34037be1dc4d42ee6c69e P aad2e8806cab4a71bcdc7e7372c36854 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } } peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } }
I20260926 20:49:19.763703 8433 leader_election.cc:288] T 4c7a47aa06c34037be1dc4d42ee6c69e P aad2e8806cab4a71bcdc7e7372c36854 [CANDIDATE]: Term 1 election: Requested vote from peers b7762c6bd65f4238ac92779f523d6b1c (127.0.98.1:38323), 5ebea5fcb4064638863a64bdbf8bb189 (127.0.98.2:43131)
I20260926 20:49:19.763865 8061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7a47aa06c34037be1dc4d42ee6c69e" candidate_uuid: "aad2e8806cab4a71bcdc7e7372c36854" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7762c6bd65f4238ac92779f523d6b1c"
I20260926 20:49:19.763886 8201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c7a47aa06c34037be1dc4d42ee6c69e" candidate_uuid: "aad2e8806cab4a71bcdc7e7372c36854" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ebea5fcb4064638863a64bdbf8bb189"
I20260926 20:49:19.763962 8061 raft_consensus.cc:3037] T 4c7a47aa06c34037be1dc4d42ee6c69e P b7762c6bd65f4238ac92779f523d6b1c [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:49:19.763983 8201 raft_consensus.cc:3037] T 4c7a47aa06c34037be1dc4d42ee6c69e P 5ebea5fcb4064638863a64bdbf8bb189 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:49:19.764544 8201 raft_consensus.cc:2454] T 4c7a47aa06c34037be1dc4d42ee6c69e P 5ebea5fcb4064638863a64bdbf8bb189 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad2e8806cab4a71bcdc7e7372c36854 in term 1.
I20260926 20:49:19.764700 8061 raft_consensus.cc:2454] T 4c7a47aa06c34037be1dc4d42ee6c69e P b7762c6bd65f4238ac92779f523d6b1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad2e8806cab4a71bcdc7e7372c36854 in term 1.
I20260926 20:49:19.764719 8272 leader_election.cc:302] T 4c7a47aa06c34037be1dc4d42ee6c69e P aad2e8806cab4a71bcdc7e7372c36854 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ebea5fcb4064638863a64bdbf8bb189, aad2e8806cab4a71bcdc7e7372c36854; no voters:
I20260926 20:49:19.764858 8433 raft_consensus.cc:2781] T 4c7a47aa06c34037be1dc4d42ee6c69e P aad2e8806cab4a71bcdc7e7372c36854 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:49:19.764911 8433 raft_consensus.cc:686] T 4c7a47aa06c34037be1dc4d42ee6c69e P aad2e8806cab4a71bcdc7e7372c36854 [term 1 LEADER]: Becoming Leader. State: Replica: aad2e8806cab4a71bcdc7e7372c36854, State: Running, Role: LEADER
I20260926 20:49:19.764961 8433 consensus_queue.cc:237] T 4c7a47aa06c34037be1dc4d42ee6c69e P aad2e8806cab4a71bcdc7e7372c36854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } } peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } }
I20260926 20:49:19.765582 7902 catalog_manager.cc:5495] T 4c7a47aa06c34037be1dc4d42ee6c69e P aad2e8806cab4a71bcdc7e7372c36854 reported cstate change: term changed from 0 to 1, leader changed from <none> to aad2e8806cab4a71bcdc7e7372c36854 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "aad2e8806cab4a71bcdc7e7372c36854" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aad2e8806cab4a71bcdc7e7372c36854" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37051 } health_report { overall_health: HEALTHY } } }
I20260926 20:49:19.775839 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:19.773322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33194 (local address 127.0.98.3:37051)
0926 20:49:19.773364 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:19.773366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:19.773377 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:49:19.773442 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:19.773443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:19.773451 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:19.773491 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:49:19.773619 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.774077 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.775113 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.775302 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.775391 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:19.775545 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:49:19.775571 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:49:19.775610 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:19.775696 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 20:49:19.775712 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 20:49:19.780380 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:19.777963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41299 (local address 127.0.98.3:37051)
0926 20:49:19.778002 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:19.778004 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:19.778013 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:49:19.778083 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:19.778084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:19.778092 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:19.778154 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:49:19.778232 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.778920 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.779682 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.780004 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.780076 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:19.780170 (+ 94us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:49:19.780197 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:49:19.780244 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:19.780277 (+ 33us) server_negotiation.cc:299] Negotiation successful
0926 20:49:19.780292 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 20:49:19.780457 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:19.777922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37051 (local address 127.0.98.3:41299)
0926 20:49:19.777973 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:19.777985 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:19.778024 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:19.778155 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:19.778156 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:19.778156 (+ 0us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:49:19.778188 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:19.778189 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.778997 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.778998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:19.779599 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:19.779601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:19.780075 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:19.780076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:19.780115 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:19.780118 (+ 3us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:49:19.780244 (+ 126us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:49:19.780245 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 20:49:19.780275 (+ 30us) client_negotiation.cc:240] Negotiation successful
0926 20:49:19.780330 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 20:49:19.782863 8061 raft_consensus.cc:1261] T 4c7a47aa06c34037be1dc4d42ee6c69e P b7762c6bd65f4238ac92779f523d6b1c [term 1 FOLLOWER]: Refusing update from remote peer aad2e8806cab4a71bcdc7e7372c36854: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:49:19.782851 8201 raft_consensus.cc:1261] T 4c7a47aa06c34037be1dc4d42ee6c69e P 5ebea5fcb4064638863a64bdbf8bb189 [term 1 FOLLOWER]: Refusing update from remote peer aad2e8806cab4a71bcdc7e7372c36854: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:49:19.783088 8433 consensus_queue.cc:1035] T 4c7a47aa06c34037be1dc4d42ee6c69e P aad2e8806cab4a71bcdc7e7372c36854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ebea5fcb4064638863a64bdbf8bb189" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:49:19.783178 8425 consensus_queue.cc:1035] T 4c7a47aa06c34037be1dc4d42ee6c69e P aad2e8806cab4a71bcdc7e7372c36854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7762c6bd65f4238ac92779f523d6b1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:49:19.793882 7902 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:49:19.793978 7902 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:49:19.794590 7902 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b012d088b8243c290a80ca96476dd0c: Sending DeleteTablet for 3 replicas of tablet 4c7a47aa06c34037be1dc4d42ee6c69e
I20260926 20:49:19.794829 8041 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c7a47aa06c34037be1dc4d42ee6c69e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:49:19 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53438
I20260926 20:49:19.794831 8320 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c7a47aa06c34037be1dc4d42ee6c69e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:49:19 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33166
I20260926 20:49:19.794874 8181 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c7a47aa06c34037be1dc4d42ee6c69e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:49:19 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48720
I20260926 20:49:19.794922 8440 tablet_replica.cc:330] T 4c7a47aa06c34037be1dc4d42ee6c69e P b7762c6bd65f4238ac92779f523d6b1c: stopping tablet replica
I20260926 20:49:19.794925 8439 tablet_replica.cc:330] T 4c7a47aa06c34037be1dc4d42ee6c69e P aad2e8806cab4a71bcdc7e7372c36854: stopping tablet replica
I20260926 20:49:19.794953 8438 tablet_replica.cc:330] T 4c7a47aa06c34037be1dc4d42ee6c69e P 5ebea5fcb4064638863a64bdbf8bb189: stopping tablet replica
I20260926 20:49:19.794963 8440 raft_consensus.cc:2229] T 4c7a47aa06c34037be1dc4d42ee6c69e P b7762c6bd65f4238ac92779f523d6b1c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:49:19.794975 8439 raft_consensus.cc:2229] T 4c7a47aa06c34037be1dc4d42ee6c69e P aad2e8806cab4a71bcdc7e7372c36854 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:49:19.794993 8440 raft_consensus.cc:2258] T 4c7a47aa06c34037be1dc4d42ee6c69e P b7762c6bd65f4238ac92779f523d6b1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:49:19.794991 8438 raft_consensus.cc:2229] T 4c7a47aa06c34037be1dc4d42ee6c69e P 5ebea5fcb4064638863a64bdbf8bb189 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:49:19.795022 8438 raft_consensus.cc:2258] T 4c7a47aa06c34037be1dc4d42ee6c69e P 5ebea5fcb4064638863a64bdbf8bb189 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:49:19.795043 8439 raft_consensus.cc:2258] T 4c7a47aa06c34037be1dc4d42ee6c69e P aad2e8806cab4a71bcdc7e7372c36854 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:49:19.795361 8439 ts_tablet_manager.cc:1905] T 4c7a47aa06c34037be1dc4d42ee6c69e P aad2e8806cab4a71bcdc7e7372c36854: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:49:19.795444 8440 ts_tablet_manager.cc:1905] T 4c7a47aa06c34037be1dc4d42ee6c69e P b7762c6bd65f4238ac92779f523d6b1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:49:19.795591 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 7976
I20260926 20:49:19.797158 8439 ts_tablet_manager.cc:1918] T 4c7a47aa06c34037be1dc4d42ee6c69e P aad2e8806cab4a71bcdc7e7372c36854: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:49:19.797226 8439 log.cc:1196] T 4c7a47aa06c34037be1dc4d42ee6c69e P aad2e8806cab4a71bcdc7e7372c36854: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-2/wal/wals/4c7a47aa06c34037be1dc4d42ee6c69e
I20260926 20:49:19.797384 8438 ts_tablet_manager.cc:1905] T 4c7a47aa06c34037be1dc4d42ee6c69e P 5ebea5fcb4064638863a64bdbf8bb189: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:49:19.797464 8439 ts_tablet_manager.cc:1939] T 4c7a47aa06c34037be1dc4d42ee6c69e P aad2e8806cab4a71bcdc7e7372c36854: Deleting consensus metadata
I20260926 20:49:19.797947 7889 catalog_manager.cc:4841] TS aad2e8806cab4a71bcdc7e7372c36854 (127.0.98.3:37051): tablet 4c7a47aa06c34037be1dc4d42ee6c69e (table test-table [id=6053e0d692bb46b8b84eb01b1bd5a821]) successfully deleted
I20260926 20:49:19.799134 8438 ts_tablet_manager.cc:1918] T 4c7a47aa06c34037be1dc4d42ee6c69e P 5ebea5fcb4064638863a64bdbf8bb189: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:49:19.799201 8438 log.cc:1196] T 4c7a47aa06c34037be1dc4d42ee6c69e P 5ebea5fcb4064638863a64bdbf8bb189: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455725523895-392-0/minicluster-data/ts-1/wal/wals/4c7a47aa06c34037be1dc4d42ee6c69e
I20260926 20:49:19.799404 8438 ts_tablet_manager.cc:1939] T 4c7a47aa06c34037be1dc4d42ee6c69e P 5ebea5fcb4064638863a64bdbf8bb189: Deleting consensus metadata
I20260926 20:49:19.799863 7888 catalog_manager.cc:4841] TS 5ebea5fcb4064638863a64bdbf8bb189 (127.0.98.2:43131): tablet 4c7a47aa06c34037be1dc4d42ee6c69e (table test-table [id=6053e0d692bb46b8b84eb01b1bd5a821]) successfully deleted
W20260926 20:49:19.802618 7888 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38323 (error 108)
I20260926 20:49:19.802963 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 8116
I20260926 20:49:19.803772 8497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:19.803036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38323 (local address 127.0.0.1:53442)
0926 20:49:19.803656 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:19.803704 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":45,"threads_started":1}
W20260926 20:49:19.803872 7888 catalog_manager.cc:4584] TS b7762c6bd65f4238ac92779f523d6b1c (127.0.98.1:38323): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4c7a47aa06c34037be1dc4d42ee6c69e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38323: connect: Connection refused (error 111)
I20260926 20:49:19.809345 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 8256
I20260926 20:49:19.816362 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 7872
2026-09-26T20:49:19Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2391 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790455725523895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:49:19 dist-test-slave-hwlj krb5kdc[8500](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:49:19 dist-test-slave-hwlj krb5kdc[8500](info): set up 2 sockets
Sep 26 20:49:19 dist-test-slave-hwlj krb5kdc[8500](info): commencing operation
krb5kdc: starting...
W20260926 20:49:20.864382 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790455725523895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790455725523895-392-0/krb5kdc/test-user.keytab.
Sep 26 20:49:20 dist-test-slave-hwlj krb5kdc[8500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:49:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:49:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 20:49:20.982484 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37503
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37503
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:49:21.062925 8513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:21.063127 8513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:21.063149 8513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:21.064316 8513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:49:21.064357 8513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:21.064373 8513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:21.064384 8513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:49:21.064442 8513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:49:21.065932 8513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790455725523895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:21.066179 8513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:21.066406 8513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:21.071434 8519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:21.071444 8521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:21.071451 8513 server_base.cc:1034] running on GCE node
W20260926 20:49:21.071437 8518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:21.071907 8513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:21.072238 8513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:21.073403 8513 hybrid_clock.cc:648] HybridClock initialized: now 1790455761073389 us; error 30 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T20:49:21Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1262 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:49:21 dist-test-slave-hwlj krb5kdc[8526](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:49:21 dist-test-slave-hwlj krb5kdc[8526](info): set up 2 sockets
Sep 26 20:49:21 dist-test-slave-hwlj krb5kdc[8526](info): commencing operation
krb5kdc: starting...
W20260926 20:49:22.108443 392 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/test-user.keytab.
Sep 26 20:49:22 dist-test-slave-hwlj krb5kdc[8526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:49:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:49:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 20:49:22.224367 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36223
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36223
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:49:22.306473 8539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:22.306712 8539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:22.306751 8539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:22.307936 8539 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:49:22.308005 8539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:22.308029 8539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:22.308053 8539 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:49:22.308081 8539 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:49:22.309605 8539 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:22.309885 8539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:22.310110 8539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:22.314842 8544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:22.314973 8539 server_base.cc:1034] running on GCE node
W20260926 20:49:22.314873 8545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:22.314864 8547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:22.315352 8539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:22.315696 8539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:22.316869 8539 hybrid_clock.cc:648] HybridClock initialized: now 1790455762316849 us; error 41 us; skew 500 ppm
Sep 26 20:49:22 dist-test-slave-hwlj krb5kdc[8526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:49:22.319223 8539 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:49:22.319710 8539 webserver.cc:460] Webserver started at http://127.0.98.62:44575/ using document root <none> and password file <none>
I20260926 20:49:22.319960 8539 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:22.320029 8539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:22.320160 8539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:49:22.321195 8539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/master-0/data/instance:
uuid: "2fafc55cfd3b4b6d8d7f17da1fac5498"
format_stamp: "Formatted at 2026-09-26 20:49:22 on dist-test-slave-hwlj"
server_key: "262b7e5a48ed7fb9f02b8f9f2c6ae1fa"
server_key_iv: "d14b59945ee58b0c720f7703aafd8e86"
server_key_version: "encryptionkey@0"
I20260926 20:49:22.321550 8539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/master-0/wal/instance:
uuid: "2fafc55cfd3b4b6d8d7f17da1fac5498"
format_stamp: "Formatted at 2026-09-26 20:49:22 on dist-test-slave-hwlj"
server_key: "262b7e5a48ed7fb9f02b8f9f2c6ae1fa"
server_key_iv: "d14b59945ee58b0c720f7703aafd8e86"
server_key_version: "encryptionkey@0"
I20260926 20:49:22.322839 8539 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:49:22.323499 8554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:22.323674 8539 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:22.323791 8539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "2fafc55cfd3b4b6d8d7f17da1fac5498"
format_stamp: "Formatted at 2026-09-26 20:49:22 on dist-test-slave-hwlj"
server_key: "262b7e5a48ed7fb9f02b8f9f2c6ae1fa"
server_key_iv: "d14b59945ee58b0c720f7703aafd8e86"
server_key_version: "encryptionkey@0"
I20260926 20:49:22.323874 8539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:22.343372 8539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:22.345436 8539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:22.345606 8539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:22.349514 8539 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36223
I20260926 20:49:22.349571 8606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36223 every 8 connection(s)
I20260926 20:49:22.350088 8539 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:49:22.350714 8607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:49:22.353490 8607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fafc55cfd3b4b6d8d7f17da1fac5498: Bootstrap starting.
I20260926 20:49:22.354084 8607 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2fafc55cfd3b4b6d8d7f17da1fac5498: Neither blocks nor log segments found. Creating new log.
I20260926 20:49:22.354393 8607 log.cc:824] T 00000000000000000000000000000000 P 2fafc55cfd3b4b6d8d7f17da1fac5498: Log is configured to *not* fsync() on all Append() calls
I20260926 20:49:22.355111 8607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fafc55cfd3b4b6d8d7f17da1fac5498: No bootstrap required, opened a new log
I20260926 20:49:22.356693 8607 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2fafc55cfd3b4b6d8d7f17da1fac5498 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fafc55cfd3b4b6d8d7f17da1fac5498" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36223 } }
I20260926 20:49:22.356834 8607 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2fafc55cfd3b4b6d8d7f17da1fac5498 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:49:22.356858 8607 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2fafc55cfd3b4b6d8d7f17da1fac5498 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fafc55cfd3b4b6d8d7f17da1fac5498, State: Initialized, Role: FOLLOWER
I20260926 20:49:22.357021 8607 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fafc55cfd3b4b6d8d7f17da1fac5498 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fafc55cfd3b4b6d8d7f17da1fac5498" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36223 } }
I20260926 20:49:22.357096 8607 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2fafc55cfd3b4b6d8d7f17da1fac5498 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:49:22.357117 8607 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2fafc55cfd3b4b6d8d7f17da1fac5498 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:49:22.357146 8607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2fafc55cfd3b4b6d8d7f17da1fac5498 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:49:22.358052 8607 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2fafc55cfd3b4b6d8d7f17da1fac5498 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fafc55cfd3b4b6d8d7f17da1fac5498" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36223 } }
I20260926 20:49:22.358134 8607 leader_election.cc:302] T 00000000000000000000000000000000 P 2fafc55cfd3b4b6d8d7f17da1fac5498 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2fafc55cfd3b4b6d8d7f17da1fac5498; no voters:
I20260926 20:49:22.358325 8607 leader_election.cc:288] T 00000000000000000000000000000000 P 2fafc55cfd3b4b6d8d7f17da1fac5498 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:49:22.358430 8610 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2fafc55cfd3b4b6d8d7f17da1fac5498 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:49:22.358597 8610 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2fafc55cfd3b4b6d8d7f17da1fac5498 [term 1 LEADER]: Becoming Leader. State: Replica: 2fafc55cfd3b4b6d8d7f17da1fac5498, State: Running, Role: LEADER
I20260926 20:49:22.358636 8607 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fafc55cfd3b4b6d8d7f17da1fac5498 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:49:22.358687 8610 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fafc55cfd3b4b6d8d7f17da1fac5498 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fafc55cfd3b4b6d8d7f17da1fac5498" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36223 } }
I20260926 20:49:22.359092 8611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fafc55cfd3b4b6d8d7f17da1fac5498 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2fafc55cfd3b4b6d8d7f17da1fac5498" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fafc55cfd3b4b6d8d7f17da1fac5498" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36223 } } }
I20260926 20:49:22.359253 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 8539
I20260926 20:49:22.359354 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/master-0/wal/instance
I20260926 20:49:22.359449 8611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fafc55cfd3b4b6d8d7f17da1fac5498 [sys.catalog]: This master's current role is: LEADER
I20260926 20:49:22.359596 392 external_mini_cluster.cc:1428] Setting key 0c01547062c75593da01a5b50640cbd0
I20260926 20:49:22.359269 8612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fafc55cfd3b4b6d8d7f17da1fac5498 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fafc55cfd3b4b6d8d7f17da1fac5498. Latest consensus state: current_term: 1 leader_uuid: "2fafc55cfd3b4b6d8d7f17da1fac5498" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fafc55cfd3b4b6d8d7f17da1fac5498" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36223 } } }
I20260926 20:49:22.359836 8612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fafc55cfd3b4b6d8d7f17da1fac5498 [sys.catalog]: This master's current role is: LEADER
W20260926 20:49:22.360473 8626 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2fafc55cfd3b4b6d8d7f17da1fac5498: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:49:22.360596 8626 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:49:22.360669 8619 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:49:22.360831 8619 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:49:22.362438 8619 catalog_manager.cc:1347] Generated new cluster ID: 1b0096cf2bf04829af9994e21ce36bbe
I20260926 20:49:22.362491 8619 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 20:49:22 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:49:22.369871 8629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:22.360336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49534 (local address 127.0.98.62:36223)
0926 20:49:22.360547 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:22.360557 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:22.360586 (+ 29us) server_negotiation.cc:407] Connection header received
0926 20:49:22.360698 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:22.360704 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:22.360758 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:22.360890 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:22.361684 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:22.362883 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:22.363649 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:22.364157 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:22.364285 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:22.366585 (+ 2300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:22.366588 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:22.366589 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:22.366600 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:22.368372 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:22.368737 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:22.368741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:22.368741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:22.368833 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:22.369041 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:22.369042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:22.369042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:22.369288 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:22.369460 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:22.369528 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 20:49:22.369663 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":166,"thread_start_us":68,"threads_started":1}
I20260926 20:49:22.381305 8619 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:49:22.382088 8619 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:49:22.390379 8619 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2fafc55cfd3b4b6d8d7f17da1fac5498: Generated new TSK 0
I20260926 20:49:22.390594 8619 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 20:49:22.422186 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36223
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:49:22.502527 8633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:22.502744 8633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:22.502784 8633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:22.503996 8633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:22.504055 8633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:22.504099 8633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:49:22.505617 8633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:22.505872 8633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:22.506098 8633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:22.510824 8641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:22.510848 8639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:22.510825 8638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:22.511212 8633 server_base.cc:1034] running on GCE node
I20260926 20:49:22.511475 8633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:22.511713 8633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:22.512897 8633 hybrid_clock.cc:648] HybridClock initialized: now 1790455762512884 us; error 34 us; skew 500 ppm
Sep 26 20:49:22 dist-test-slave-hwlj krb5kdc[8526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:49:22.515349 8633 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 20:49:22.515900 8633 webserver.cc:460] Webserver started at http://127.0.98.1:32891/ using document root <none> and password file <none>
I20260926 20:49:22.516146 8633 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:22.516212 8633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:22.516367 8633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:49:22.517400 8633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-0/data/instance:
uuid: "06a02ee3cfd54c78b8f4596a2373f6b6"
format_stamp: "Formatted at 2026-09-26 20:49:22 on dist-test-slave-hwlj"
server_key: "574443f616650535018941a83e0545d6"
server_key_iv: "885f8c1f27fe48fa8a396bace97b8bec"
server_key_version: "encryptionkey@0"
I20260926 20:49:22.517759 8633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-0/wal/instance:
uuid: "06a02ee3cfd54c78b8f4596a2373f6b6"
format_stamp: "Formatted at 2026-09-26 20:49:22 on dist-test-slave-hwlj"
server_key: "574443f616650535018941a83e0545d6"
server_key_iv: "885f8c1f27fe48fa8a396bace97b8bec"
server_key_version: "encryptionkey@0"
I20260926 20:49:22.519004 8633 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 20:49:22.519696 8648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:22.519879 8633 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:22.519975 8633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-0/wal
uuid: "06a02ee3cfd54c78b8f4596a2373f6b6"
format_stamp: "Formatted at 2026-09-26 20:49:22 on dist-test-slave-hwlj"
server_key: "574443f616650535018941a83e0545d6"
server_key_iv: "885f8c1f27fe48fa8a396bace97b8bec"
server_key_version: "encryptionkey@0"
I20260926 20:49:22.520061 8633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:22.530516 8633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:22.532891 8633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:22.533051 8633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:22.533272 8633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:49:22.533578 8633 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:49:22.533631 8633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:22.533674 8633 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:49:22.533699 8633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:22.539500 8633 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45877
I20260926 20:49:22.539542 8761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45877 every 8 connection(s)
I20260926 20:49:22.539911 8633 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 20:49:22 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:49:22.547219 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 8633
I20260926 20:49:22.547355 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-0/wal/instance
I20260926 20:49:22.547557 392 external_mini_cluster.cc:1428] Setting key 7d6e69dc3c4f2f1f2ba36b82142f6ffc
I20260926 20:49:22.548906 8629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:22.540628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36127 (local address 127.0.98.62:36223)
0926 20:49:22.540699 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:22.540701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:22.540906 (+ 205us) server_negotiation.cc:407] Connection header received
0926 20:49:22.541802 (+ 896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:22.541803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:22.541813 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:22.541871 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:22.542433 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:22.542929 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:22.543894 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:22.544097 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:22.544149 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:22.546119 (+ 1970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:22.546121 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:22.546121 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:22.546127 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:22.547608 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:22.548014 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:22.548015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:22.548016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:22.548075 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:22.548249 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:22.548249 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:22.548250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:22.548387 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:22.548439 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:22.548770 (+ 331us) server_negotiation.cc:299] Negotiation successful
0926 20:49:22.548839 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 20:49:22.549315 8764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:22.540612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.1:36127)
0926 20:49:22.540778 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:22.540820 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:22.541707 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:22.541930 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:22.541933 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:22.542177 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:22.542364 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:22.542367 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:22.543005 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:22.543006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:22.543825 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:22.543827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:22.544167 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:22.544167 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:22.544258 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:22.544682 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:22.544690 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:22.546033 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:22.547692 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:22.547699 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:22.547701 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:22.547948 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:22.548129 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:22.548129 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:22.548129 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:22.548210 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:22.548457 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:22.548462 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:22.548630 (+ 168us) client_negotiation.cc:769] Sending connection context
0926 20:49:22.548749 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 20:49:22.548923 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":93,"threads_started":1}
I20260926 20:49:22.549842 8762 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36223
I20260926 20:49:22.549960 8762 heartbeater.cc:464] Registering TS with master...
I20260926 20:49:22.550164 8762 heartbeater.cc:511] Master 127.0.98.62:36223 requested a full tablet report, sending...
I20260926 20:49:22.550683 8571 ts_manager.cc:194] Registered new tserver with Master: 06a02ee3cfd54c78b8f4596a2373f6b6 (127.0.98.1:45877)
I20260926 20:49:22.551312 8571 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36127
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 20:49:22.574894 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36223
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:49:22.656052 8767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:22.656243 8767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:22.656272 8767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:22.657418 8767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:22.657459 8767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:22.657516 8767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:49:22.659128 8767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:22.659366 8767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:22.659587 8767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:22.664551 8775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:22.664577 8773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:22.664547 8772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:22.664781 8767 server_base.cc:1034] running on GCE node
I20260926 20:49:22.665002 8767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:22.665252 8767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:22.666417 8767 hybrid_clock.cc:648] HybridClock initialized: now 1790455762666405 us; error 28 us; skew 500 ppm
Sep 26 20:49:22 dist-test-slave-hwlj krb5kdc[8526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:49:22.669020 8767 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 20:49:22.669453 8767 webserver.cc:460] Webserver started at http://127.0.98.2:39103/ using document root <none> and password file <none>
I20260926 20:49:22.669687 8767 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:22.669734 8767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:22.669860 8767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:49:22.670876 8767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-1/data/instance:
uuid: "ad2a2bf01b974411aa1a26b1cfe868fe"
format_stamp: "Formatted at 2026-09-26 20:49:22 on dist-test-slave-hwlj"
server_key: "6c8853d3ea729e9e7f340f77f1c15298"
server_key_iv: "dd4c1274896ec211d27a9e1cd5014de5"
server_key_version: "encryptionkey@0"
I20260926 20:49:22.671221 8767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ad2a2bf01b974411aa1a26b1cfe868fe"
format_stamp: "Formatted at 2026-09-26 20:49:22 on dist-test-slave-hwlj"
server_key: "6c8853d3ea729e9e7f340f77f1c15298"
server_key_iv: "dd4c1274896ec211d27a9e1cd5014de5"
server_key_version: "encryptionkey@0"
I20260926 20:49:22.672626 8767 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:49:22.673370 8782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:22.673560 8767 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:49:22.673664 8767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-1/wal
uuid: "ad2a2bf01b974411aa1a26b1cfe868fe"
format_stamp: "Formatted at 2026-09-26 20:49:22 on dist-test-slave-hwlj"
server_key: "6c8853d3ea729e9e7f340f77f1c15298"
server_key_iv: "dd4c1274896ec211d27a9e1cd5014de5"
server_key_version: "encryptionkey@0"
I20260926 20:49:22.673738 8767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:22.695505 8767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:22.697992 8767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:22.698168 8767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:22.698426 8767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:49:22.698748 8767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:49:22.698803 8767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:22.698848 8767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:49:22.698872 8767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:22.704598 8767 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32925
I20260926 20:49:22.704676 8895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32925 every 8 connection(s)
I20260926 20:49:22.705004 8767 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
I20260926 20:49:22.710196 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 8767
I20260926 20:49:22.710292 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-1/wal/instance
I20260926 20:49:22.710517 392 external_mini_cluster.cc:1428] Setting key 46a279f9c058b4b4551e255ddbeb78b2
Sep 26 20:49:22 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:49:22.714859 8629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:22.705820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56147 (local address 127.0.98.62:36223)
0926 20:49:22.705911 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:22.705913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:22.706071 (+ 158us) server_negotiation.cc:407] Connection header received
0926 20:49:22.707001 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:22.707002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:22.707013 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:22.707074 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:22.707838 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:22.708321 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:22.709307 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:22.709506 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:22.709561 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:22.712031 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:22.712032 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:22.712033 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:22.712039 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:22.713450 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:22.713901 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:22.713902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:22.713903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:22.713975 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:22.714201 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:22.714201 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:22.714202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:22.714353 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:22.714427 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:22.714651 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 20:49:22.714745 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 20:49:22.715000 8898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:22.705775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.2:56147)
0926 20:49:22.705940 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:22.705982 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:22.706883 (+ 901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:22.707130 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:22.707134 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:22.707466 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:22.707696 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:22.707700 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:22.708434 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:22.708435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:22.709227 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:22.709229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:22.709577 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:22.709578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:22.709674 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:22.710085 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:22.710093 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:22.711948 (+ 1855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:22.713555 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:22.713568 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:22.713570 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:22.713826 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:22.714043 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:22.714043 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:22.714043 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:22.714139 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:22.714413 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:22.714416 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:22.714527 (+ 111us) client_negotiation.cc:769] Sending connection context
0926 20:49:22.714605 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 20:49:22.714724 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":94,"threads_started":1}
I20260926 20:49:22.715535 8896 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36223
I20260926 20:49:22.715638 8896 heartbeater.cc:464] Registering TS with master...
I20260926 20:49:22.715878 8896 heartbeater.cc:511] Master 127.0.98.62:36223 requested a full tablet report, sending...
I20260926 20:49:22.716300 8571 ts_manager.cc:194] Registered new tserver with Master: ad2a2bf01b974411aa1a26b1cfe868fe (127.0.98.2:32925)
I20260926 20:49:22.716691 8571 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56147
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 20:49:22.739717 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36223
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:49:22.821820 8901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:22.822118 8901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:22.822163 8901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:22.823571 8901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:22.823634 8901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:22.823681 8901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:49:22.825405 8901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:22.825685 8901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:22.825933 8901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:22.831260 8906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:22.831267 8909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:22.831274 8907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:22.832041 8901 server_base.cc:1034] running on GCE node
I20260926 20:49:22.832355 8901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:22.832628 8901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:22.833793 8901 hybrid_clock.cc:648] HybridClock initialized: now 1790455762833776 us; error 32 us; skew 500 ppm
Sep 26 20:49:22 dist-test-slave-hwlj krb5kdc[8526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:49:22.836561 8901 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 20:49:22.836977 8901 webserver.cc:460] Webserver started at http://127.0.98.3:46407/ using document root <none> and password file <none>
I20260926 20:49:22.837185 8901 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:22.837229 8901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:22.837360 8901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:49:22.838462 8901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-2/data/instance:
uuid: "d8e3d6773e6e475aa1621ea3adcf99c4"
format_stamp: "Formatted at 2026-09-26 20:49:22 on dist-test-slave-hwlj"
server_key: "040303b37396142b65fa57524c92feb2"
server_key_iv: "9dd0ddce29ea1691d2ac0ca176053f8f"
server_key_version: "encryptionkey@0"
I20260926 20:49:22.838776 8901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d8e3d6773e6e475aa1621ea3adcf99c4"
format_stamp: "Formatted at 2026-09-26 20:49:22 on dist-test-slave-hwlj"
server_key: "040303b37396142b65fa57524c92feb2"
server_key_iv: "9dd0ddce29ea1691d2ac0ca176053f8f"
server_key_version: "encryptionkey@0"
I20260926 20:49:22.840243 8901 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:49:22.841015 8916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:22.841226 8901 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:49:22.841315 8901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-2/wal
uuid: "d8e3d6773e6e475aa1621ea3adcf99c4"
format_stamp: "Formatted at 2026-09-26 20:49:22 on dist-test-slave-hwlj"
server_key: "040303b37396142b65fa57524c92feb2"
server_key_iv: "9dd0ddce29ea1691d2ac0ca176053f8f"
server_key_version: "encryptionkey@0"
I20260926 20:49:22.841394 8901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:22.854291 8901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:22.856365 8901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:22.856487 8901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:22.856667 8901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:49:22.856993 8901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:49:22.857028 8901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:22.857072 8901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:49:22.857100 8901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:22.863585 8901 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38405
I20260926 20:49:22.863628 9029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38405 every 8 connection(s)
I20260926 20:49:22.863986 8901 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
I20260926 20:49:22.864675 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 8901
I20260926 20:49:22.864789 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-2/wal/instance
I20260926 20:49:22.865003 392 external_mini_cluster.cc:1428] Setting key 2e29299959bc3e014fd07d7866b8d498
Sep 26 20:49:22 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:49:22.874037 8629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:22.864727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54355 (local address 127.0.98.62:36223)
0926 20:49:22.864797 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:22.864799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:22.865033 (+ 234us) server_negotiation.cc:407] Connection header received
0926 20:49:22.866315 (+ 1282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:22.866316 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:22.866327 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:22.866376 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:22.866971 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:22.867459 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:22.868690 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:22.868869 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:22.868925 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:22.871296 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:22.871298 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:22.871298 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:22.871304 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:22.872750 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:22.873174 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:22.873174 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:22.873175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:22.873221 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:22.873464 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:22.873465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:22.873465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:22.873568 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:22.873619 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:22.873914 (+ 295us) server_negotiation.cc:299] Negotiation successful
0926 20:49:22.873973 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 20:49:22.874259 9032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:22.864694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.3:54355)
0926 20:49:22.864877 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:22.864930 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:22.866217 (+ 1287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:22.866433 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:22.866437 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:22.866693 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:22.866899 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:22.866904 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:22.867563 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:22.867564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:22.868614 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:22.868617 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:22.868949 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:22.868950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:22.869061 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:22.869545 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:22.869553 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:22.871207 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:22.872858 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:22.872863 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:22.872865 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:22.873103 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:22.873282 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:22.873283 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:22.873283 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:22.873400 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:22.873653 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:22.873658 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:22.873785 (+ 127us) client_negotiation.cc:769] Sending connection context
0926 20:49:22.873889 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 20:49:22.874019 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":78,"threads_started":1}
I20260926 20:49:22.874929 9030 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36223
I20260926 20:49:22.875044 9030 heartbeater.cc:464] Registering TS with master...
I20260926 20:49:22.875240 9030 heartbeater.cc:511] Master 127.0.98.62:36223 requested a full tablet report, sending...
I20260926 20:49:22.875622 8571 ts_manager.cc:194] Registered new tserver with Master: d8e3d6773e6e475aa1621ea3adcf99c4 (127.0.98.3:38405)
I20260926 20:49:22.876039 8571 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54355
I20260926 20:49:22.876791 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260926 20:49:22.954484 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 8633
I20260926 20:49:22.959051 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 8767
I20260926 20:49:22.963553 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 8901
I20260926 20:49:22.968132 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 8539
I20260926 20:49:22.973193 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36223
--webserver_interface=127.0.98.62
--webserver_port=44575
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36223
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:49:23.052726 9039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:23.052973 9039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:23.053011 9039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:23.054134 9039 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:49:23.054203 9039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:23.054225 9039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:23.054246 9039 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:49:23.054276 9039 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:49:23.055733 9039 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=44575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:23.056038 9039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:23.056265 9039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:23.060863 9044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:23.060889 9045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:23.060863 9047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:23.061420 9039 server_base.cc:1034] running on GCE node
I20260926 20:49:23.061646 9039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:23.061880 9039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:23.063043 9039 hybrid_clock.cc:648] HybridClock initialized: now 1790455763063026 us; error 30 us; skew 500 ppm
Sep 26 20:49:23 dist-test-slave-hwlj krb5kdc[8526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:49:23.065755 9039 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:49:23.066184 9039 webserver.cc:460] Webserver started at http://127.0.98.62:44575/ using document root <none> and password file <none>
I20260926 20:49:23.066409 9039 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:23.066465 9039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:23.067907 9039 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:49:23.068544 9054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:23.068714 9039 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:23.068810 9039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "2fafc55cfd3b4b6d8d7f17da1fac5498"
format_stamp: "Formatted at 2026-09-26 20:49:22 on dist-test-slave-hwlj"
server_key: "262b7e5a48ed7fb9f02b8f9f2c6ae1fa"
server_key_iv: "d14b59945ee58b0c720f7703aafd8e86"
server_key_version: "encryptionkey@0"
I20260926 20:49:23.069159 9039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:23.085891 9039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:23.088004 9039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:23.088173 9039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:23.091853 9039 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36223
I20260926 20:49:23.091894 9106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36223 every 8 connection(s)
I20260926 20:49:23.092432 9039 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:49:23.093637 9107 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:49:23.094277 9107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fafc55cfd3b4b6d8d7f17da1fac5498: Bootstrap starting.
I20260926 20:49:23.097684 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 9039
I20260926 20:49:23.098039 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45877
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36223
--webserver_port=32891
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:49:23.102873 9107 log.cc:824] T 00000000000000000000000000000000 P 2fafc55cfd3b4b6d8d7f17da1fac5498: Log is configured to *not* fsync() on all Append() calls
I20260926 20:49:23.104607 9107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fafc55cfd3b4b6d8d7f17da1fac5498: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:49:23.104889 9107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fafc55cfd3b4b6d8d7f17da1fac5498: Bootstrap complete.
I20260926 20:49:23.107138 9107 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2fafc55cfd3b4b6d8d7f17da1fac5498 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fafc55cfd3b4b6d8d7f17da1fac5498" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36223 } }
I20260926 20:49:23.107389 9107 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2fafc55cfd3b4b6d8d7f17da1fac5498 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fafc55cfd3b4b6d8d7f17da1fac5498, State: Initialized, Role: FOLLOWER
I20260926 20:49:23.107595 9107 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fafc55cfd3b4b6d8d7f17da1fac5498 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fafc55cfd3b4b6d8d7f17da1fac5498" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36223 } }
I20260926 20:49:23.107690 9107 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2fafc55cfd3b4b6d8d7f17da1fac5498 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:49:23.107765 9107 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2fafc55cfd3b4b6d8d7f17da1fac5498 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:49:23.107820 9107 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2fafc55cfd3b4b6d8d7f17da1fac5498 [term 1 FOLLOWER]: Advancing to term 2
I20260926 20:49:23.108809 9107 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2fafc55cfd3b4b6d8d7f17da1fac5498 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fafc55cfd3b4b6d8d7f17da1fac5498" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36223 } }
I20260926 20:49:23.108929 9107 leader_election.cc:302] T 00000000000000000000000000000000 P 2fafc55cfd3b4b6d8d7f17da1fac5498 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2fafc55cfd3b4b6d8d7f17da1fac5498; no voters:
I20260926 20:49:23.109125 9107 leader_election.cc:288] T 00000000000000000000000000000000 P 2fafc55cfd3b4b6d8d7f17da1fac5498 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 20:49:23.109163 9111 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2fafc55cfd3b4b6d8d7f17da1fac5498 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 20:49:23.109267 9111 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2fafc55cfd3b4b6d8d7f17da1fac5498 [term 2 LEADER]: Becoming Leader. State: Replica: 2fafc55cfd3b4b6d8d7f17da1fac5498, State: Running, Role: LEADER
I20260926 20:49:23.109362 9111 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fafc55cfd3b4b6d8d7f17da1fac5498 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fafc55cfd3b4b6d8d7f17da1fac5498" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36223 } }
I20260926 20:49:23.109438 9107 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fafc55cfd3b4b6d8d7f17da1fac5498 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:49:23.109654 9111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fafc55cfd3b4b6d8d7f17da1fac5498 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fafc55cfd3b4b6d8d7f17da1fac5498. Latest consensus state: current_term: 2 leader_uuid: "2fafc55cfd3b4b6d8d7f17da1fac5498" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fafc55cfd3b4b6d8d7f17da1fac5498" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36223 } } }
I20260926 20:49:23.109720 9111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fafc55cfd3b4b6d8d7f17da1fac5498 [sys.catalog]: This master's current role is: LEADER
I20260926 20:49:23.109843 9112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fafc55cfd3b4b6d8d7f17da1fac5498 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2fafc55cfd3b4b6d8d7f17da1fac5498" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fafc55cfd3b4b6d8d7f17da1fac5498" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36223 } } }
I20260926 20:49:23.109966 9112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fafc55cfd3b4b6d8d7f17da1fac5498 [sys.catalog]: This master's current role is: LEADER
I20260926 20:49:23.110647 9115 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:49:23.111239 9115 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:49:23.111362 9115 catalog_manager.cc:1259] Loaded cluster ID: 1b0096cf2bf04829af9994e21ce36bbe
I20260926 20:49:23.111397 9115 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:49:23.112655 9115 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:49:23.112834 9115 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2fafc55cfd3b4b6d8d7f17da1fac5498: Loaded TSK: 0
I20260926 20:49:23.113046 9115 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 20:49:23.180864 9109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:23.181066 9109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:23.181089 9109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:23.182256 9109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:23.182298 9109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:23.182324 9109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:49:23.183804 9109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45877
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=32891
--tserver_master_addrs=127.0.98.62:36223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:23.184038 9109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:23.184262 9109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:23.189096 9134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:23.189096 9133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:23.189172 9109 server_base.cc:1034] running on GCE node
W20260926 20:49:23.189108 9136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:23.189579 9109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:23.189854 9109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:23.191038 9109 hybrid_clock.cc:648] HybridClock initialized: now 1790455763191023 us; error 28 us; skew 500 ppm
Sep 26 20:49:23 dist-test-slave-hwlj krb5kdc[8526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455763, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:49:23.194000 9109 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260926 20:49:23.194474 9109 webserver.cc:460] Webserver started at http://127.0.98.1:32891/ using document root <none> and password file <none>
I20260926 20:49:23.194715 9109 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:23.194799 9109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:23.196240 9109 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:49:23.196902 9143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:23.197095 9109 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:49:23.197201 9109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-0/wal
uuid: "06a02ee3cfd54c78b8f4596a2373f6b6"
format_stamp: "Formatted at 2026-09-26 20:49:22 on dist-test-slave-hwlj"
server_key: "574443f616650535018941a83e0545d6"
server_key_iv: "885f8c1f27fe48fa8a396bace97b8bec"
server_key_version: "encryptionkey@0"
I20260926 20:49:23.197559 9109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:23.211165 9109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:23.213640 9109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:23.213816 9109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:23.214077 9109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:49:23.214388 9109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:49:23.214442 9109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:23.214483 9109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:49:23.214526 9109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:23.220594 9109 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45877
I20260926 20:49:23.220641 9256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45877 every 8 connection(s)
I20260926 20:49:23.221061 9109 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:49:23.223368 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 9109
I20260926 20:49:23.223729 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:32925
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36223
--webserver_port=39103
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:49:23 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:49:23 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:49:23.232734 9259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:23.221950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.1:60549)
0926 20:49:23.222149 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:23.222197 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:23.222687 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:23.223130 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:23.223133 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:23.224087 (+ 954us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:23.224385 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:23.224392 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:23.226096 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:23.226097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:23.227517 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:23.227521 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:23.228438 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:23.228440 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:23.228612 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:23.229256 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:23.229267 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:23.232461 (+ 3194us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":91,"threads_started":1}
W20260926 20:49:23.233139 9260 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:23.222020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60549 (local address 127.0.98.62:36223)
0926 20:49:23.222222 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:23.222232 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:23.222295 (+ 63us) server_negotiation.cc:407] Connection header received
0926 20:49:23.222880 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:23.222887 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:23.222945 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:23.223086 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:23.224537 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:23.225976 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:23.227620 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:23.228321 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:23.228456 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:23.232645 (+ 4189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60549: BlockingRecv error: recv got EOF from 127.0.98.1:60549 (error 108)
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":98,"threads_started":1}
W20260926 20:49:23.233203 9257 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36223 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:36223: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 26 20:49:23 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:49:23 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:49:23.239055 9259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:23.233640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.1:47605)
0926 20:49:23.233682 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:23.233693 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:23.233733 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:23.233964 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:23.233965 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:23.234135 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:23.234199 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:23.234201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:23.235119 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:23.235121 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:23.235991 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:23.235993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:23.236455 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:23.236456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:23.236526 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:23.237003 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:23.237008 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:23.238988 (+ 1980us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260926 20:49:23.239157 9260 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:23.233631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47605 (local address 127.0.98.62:36223)
0926 20:49:23.233853 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:23.233855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:23.233862 (+ 7us) server_negotiation.cc:407] Connection header received
0926 20:49:23.233885 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:23.233886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:23.233898 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:23.233947 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:23.234286 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:23.235036 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:23.236081 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:23.236381 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:23.236447 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:23.239092 (+ 2645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47605: BlockingRecv error: recv got EOF from 127.0.98.1:47605 (error 108)
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":90,"threads_started":1}
Sep 26 20:49:23 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:49:23 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:49:23.244510 9259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:23.239518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.1:35541)
0926 20:49:23.239560 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:23.239570 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:23.239603 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:23.239733 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:23.239734 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:23.239884 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:23.239921 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:23.239923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:23.240854 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:23.240855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:23.241622 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:23.241626 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:23.242094 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:23.242095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:23.242135 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:23.242634 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:23.242638 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:23.244457 (+ 1819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260926 20:49:23.244660 9257 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 20:49:23.244767 9260 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:23.239575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35541 (local address 127.0.98.62:36223)
0926 20:49:23.239608 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:23.239609 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:23.239620 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:49:23.239645 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:23.239646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:23.239655 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:23.239714 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:23.239992 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:23.240757 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:23.241714 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:23.242014 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:23.242086 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:23.244564 (+ 2478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35541: BlockingRecv error: recv got EOF from 127.0.98.1:35541 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 20:49:23.313336 9261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:23.313560 9261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:23.313585 9261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:23.314702 9261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:23.314741 9261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:23.314767 9261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:49:23.316223 9261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:32925
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39103
--tserver_master_addrs=127.0.98.62:36223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:23.316411 9261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:23.316586 9261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:23.321573 9268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:23.321581 9270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:23.321578 9267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:23.321780 9261 server_base.cc:1034] running on GCE node
I20260926 20:49:23.322003 9261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:23.322264 9261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:23.323437 9261 hybrid_clock.cc:648] HybridClock initialized: now 1790455763323412 us; error 39 us; skew 500 ppm
Sep 26 20:49:23 dist-test-slave-hwlj krb5kdc[8526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455763, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:49:23.326143 9261 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260926 20:49:23.326602 9261 webserver.cc:460] Webserver started at http://127.0.98.2:39103/ using document root <none> and password file <none>
I20260926 20:49:23.326781 9261 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:23.326831 9261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:23.328205 9261 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:49:23.328845 9277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:23.329111 9261 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:49:23.329202 9261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-1/wal
uuid: "ad2a2bf01b974411aa1a26b1cfe868fe"
format_stamp: "Formatted at 2026-09-26 20:49:22 on dist-test-slave-hwlj"
server_key: "6c8853d3ea729e9e7f340f77f1c15298"
server_key_iv: "dd4c1274896ec211d27a9e1cd5014de5"
server_key_version: "encryptionkey@0"
I20260926 20:49:23.329578 9261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:23.348031 9261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:23.350365 9261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:23.350569 9261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:23.350803 9261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:49:23.351120 9261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:49:23.351174 9261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:23.351215 9261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:49:23.351243 9261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:23.357036 9261 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32925
I20260926 20:49:23.357075 9390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32925 every 8 connection(s)
I20260926 20:49:23.357403 9261 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
I20260926 20:49:23.359189 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 9261
I20260926 20:49:23.359500 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:38405
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36223
--webserver_port=46407
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:49:23 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:49:23 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:49:23.365665 9393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:23.358052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.2:32983)
0926 20:49:23.358207 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:23.358248 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:23.358702 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:23.358947 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:23.358950 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:23.359646 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:23.359944 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:23.359949 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:23.360939 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:23.360940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:23.362178 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:23.362180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:23.362685 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:23.362686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:23.362794 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:23.363353 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:23.363361 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:23.365528 (+ 2167us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":91,"threads_started":1}
W20260926 20:49:23.365734 9260 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:23.358080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32983 (local address 127.0.98.62:36223)
0926 20:49:23.358127 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:23.358129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:23.358337 (+ 208us) server_negotiation.cc:407] Connection header received
0926 20:49:23.358805 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:23.358805 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:23.358819 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:23.358877 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:23.360045 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:23.360820 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:23.362268 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:23.362601 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:23.362679 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:23.365650 (+ 2971us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:32983: BlockingRecv error: recv got EOF from 127.0.98.2:32983 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 20:49:23.366004 9391 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36223 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:36223: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 26 20:49:23 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:49:23 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:49:23.370512 9393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:23.366279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.2:56659)
0926 20:49:23.366312 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:23.366323 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:23.366384 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:23.366512 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:23.366512 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:23.366640 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:23.366683 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:23.366684 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:23.367355 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:23.367357 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:23.367934 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:23.367936 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:23.368317 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:23.368317 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:23.368380 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:23.368824 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:23.368829 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:23.370432 (+ 1603us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":16}
W20260926 20:49:23.370622 9260 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:23.366319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56659 (local address 127.0.98.62:36223)
0926 20:49:23.366378 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:23.366380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:23.366389 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:49:23.366449 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:23.366450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:23.366463 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:23.366511 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:23.366764 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:23.367254 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:23.368015 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:23.368220 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:23.368322 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:23.370508 (+ 2186us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56659: BlockingRecv error: recv got EOF from 127.0.98.2:56659 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 26 20:49:23 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:49:23 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:49:23.375387 9393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:23.370762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.2:34533)
0926 20:49:23.370899 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:23.370910 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:23.370965 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:23.371103 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:23.371104 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:23.371246 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:23.371279 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:23.371280 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:23.371957 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:23.371958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:23.372751 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:23.372753 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:23.373139 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:23.373140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:23.373228 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:23.373609 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:23.373614 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:23.375341 (+ 1727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113}
W20260926 20:49:23.375536 9262 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:23.370824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34533 (local address 127.0.98.62:36223)
0926 20:49:23.370880 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:23.370882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:23.370954 (+ 72us) server_negotiation.cc:407] Connection header received
0926 20:49:23.371012 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:23.371013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:23.371022 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:23.371081 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:23.371343 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:23.371875 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:23.372832 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:23.373059 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:23.373132 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:23.375458 (+ 2326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34533: BlockingRecv error: recv got EOF from 127.0.98.2:34533 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 20:49:23.375603 9391 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 20:49:23.442111 9394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:23.442349 9394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:23.442387 9394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:23.443557 9394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:23.443629 9394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:23.443672 9394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:49:23.445127 9394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:38405
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=46407
--tserver_master_addrs=127.0.98.62:36223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:23.445401 9394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:23.445632 9394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:23.450609 9399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:23.450616 9400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:23.450722 9394 server_base.cc:1034] running on GCE node
W20260926 20:49:23.450616 9402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:23.450999 9394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:23.451211 9394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:23.452382 9394 hybrid_clock.cc:648] HybridClock initialized: now 1790455763452362 us; error 33 us; skew 500 ppm
Sep 26 20:49:23 dist-test-slave-hwlj krb5kdc[8526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455763, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:49:23.454975 9394 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260926 20:49:23.455395 9394 webserver.cc:460] Webserver started at http://127.0.98.3:46407/ using document root <none> and password file <none>
I20260926 20:49:23.455591 9394 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:23.455641 9394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:23.457023 9394 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:49:23.457592 9409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:23.457898 9394 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:49:23.458009 9394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-2/wal
uuid: "d8e3d6773e6e475aa1621ea3adcf99c4"
format_stamp: "Formatted at 2026-09-26 20:49:22 on dist-test-slave-hwlj"
server_key: "040303b37396142b65fa57524c92feb2"
server_key_iv: "9dd0ddce29ea1691d2ac0ca176053f8f"
server_key_version: "encryptionkey@0"
I20260926 20:49:23.458397 9394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:23.477103 9394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:23.479482 9394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:23.479651 9394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:23.479923 9394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:49:23.480235 9394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:49:23.480289 9394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:23.480331 9394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:49:23.480360 9394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:23.485998 9394 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38405
I20260926 20:49:23.486047 9522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38405 every 8 connection(s)
I20260926 20:49:23.486379 9394 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 20:49:23 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:49:23 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:49:23.493417 9525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:23.487018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.3:58685)
0926 20:49:23.487201 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:23.487246 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:23.487707 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:23.487979 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:23.487982 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:23.488622 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:23.488822 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:23.488827 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:23.489459 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:23.489460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:23.490273 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:23.490275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:23.490594 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:23.490595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:23.490687 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:23.491242 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:23.491251 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:23.493281 (+ 2030us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":107,"threads_started":1}
W20260926 20:49:23.493482 9262 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:23.487032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58685 (local address 127.0.98.62:36223)
0926 20:49:23.487118 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:23.487120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:23.487339 (+ 219us) server_negotiation.cc:407] Connection header received
0926 20:49:23.487850 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:23.487851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:23.487863 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:23.487927 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:23.488905 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:23.489380 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:23.490346 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:23.490528 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:23.490577 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:23.493404 (+ 2827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58685: BlockingRecv error: recv got EOF from 127.0.98.3:58685 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260926 20:49:23.493626 9523 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36223 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:36223: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260926 20:49:23.495038 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 9394
Sep 26 20:49:23 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:49:23 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:49:23.498543 9525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:23.493856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.3:46947)
0926 20:49:23.493908 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:23.493919 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:23.493958 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:23.494217 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:23.494218 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:23.494365 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:23.494408 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:23.494410 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:23.495058 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:23.495059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:23.495844 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:23.495846 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:23.496213 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:23.496214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:23.496278 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:23.496673 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:23.496678 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:23.498452 (+ 1774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 20:49:23.498611 9262 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:23.494040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46947 (local address 127.0.98.62:36223)
0926 20:49:23.494092 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:23.494094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:23.494100 (+ 6us) server_negotiation.cc:407] Connection header received
0926 20:49:23.494117 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:23.494118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:23.494127 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:23.494178 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:23.494482 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:23.494946 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:23.495927 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:23.496114 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:23.496200 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:23.498546 (+ 2346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46947: BlockingRecv error: recv got EOF from 127.0.98.3:46947 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 26 20:49:23 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:49:23 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:49:23.503281 9260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:23.495448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49550 (local address 127.0.98.62:36223)
0926 20:49:23.495488 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:23.495490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:23.495628 (+ 138us) server_negotiation.cc:407] Connection header received
0926 20:49:23.495677 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:23.495678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:23.495687 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:23.495732 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:23.496306 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:23.496984 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:23.497986 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:23.498154 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:23.498236 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:23.499740 (+ 1504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:23.499780 (+ 40us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:23.499780 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:23.499793 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:23.501751 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:23.502111 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:23.502114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:23.502115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:23.502212 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:23.502465 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:23.502466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:23.502466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:23.502775 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:23.502878 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:23.503055 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 20:49:23.503180 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 20:49:23.503494 9525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:23.498932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.3:59753)
0926 20:49:23.498981 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:23.498991 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:23.499026 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:23.499142 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:23.499142 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:23.499244 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:23.499268 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:23.499269 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:23.500096 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:23.500097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:23.500896 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:23.500899 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:23.501223 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:23.501224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:23.501276 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:23.501572 (+ 296us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:23.501577 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:23.503436 (+ 1859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 20:49:23.503600 9523 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 20:49:23.503942 9262 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:23.498994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59753 (local address 127.0.98.62:36223)
0926 20:49:23.499029 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:23.499031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:23.499040 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:49:23.499061 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:23.499062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:23.499071 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:23.499141 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:23.499361 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:23.500022 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:23.500972 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:23.501150 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:23.501203 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:23.503882 (+ 2679us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59753: BlockingRecv error: recv got EOF from 127.0.98.3:59753 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 26 20:49:24 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:49:24 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:49:24.253934 9527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:24.245086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.1:39377)
0926 20:49:24.245335 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:24.245345 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:24.245395 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:24.245528 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:24.245528 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:24.245689 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:24.245742 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:24.245743 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:24.246431 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:24.246432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:24.246991 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:24.246993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:24.247362 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:24.247363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:24.247407 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:24.251652 (+ 4245us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:24.251658 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:24.253862 (+ 2204us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":185,"threads_started":1}
W20260926 20:49:24.254089 9528 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:24.245255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39377 (local address 127.0.98.62:36223)
0926 20:49:24.245399 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:24.245401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:24.245410 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:49:24.245434 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:24.245434 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:24.245444 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:24.245509 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:24.245815 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:24.246350 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:24.247086 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:24.247285 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:24.247349 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:24.253984 (+ 6635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39377: BlockingRecv error: recv got EOF from 127.0.98.1:39377 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":94,"threads_started":1}
Sep 26 20:49:24 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:49:24 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:49:24.380566 9529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:24.376017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.2:37253)
0926 20:49:24.376170 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:24.376182 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:24.376247 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:24.376390 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:24.376391 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:24.376551 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:24.376601 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:24.376602 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:24.377244 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:24.377244 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:24.377773 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:24.377774 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:24.378097 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:24.378097 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:24.378134 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:24.378671 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:24.378676 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:24.380497 (+ 1821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":101,"threads_started":1}
W20260926 20:49:24.380635 9528 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:24.376099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37253 (local address 127.0.98.62:36223)
0926 20:49:24.376195 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:24.376197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:24.376236 (+ 39us) server_negotiation.cc:407] Connection header received
0926 20:49:24.376310 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:24.376310 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:24.376322 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:24.376366 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:24.376694 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:24.377174 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:24.377848 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:24.378033 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:24.378085 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:24.380589 (+ 2504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37253: BlockingRecv error: recv got EOF from 127.0.98.2:37253 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 26 20:49:24 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:49:24 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:49:24.508496 9530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:24.503978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.3:34675)
0926 20:49:24.504113 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:24.504123 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:24.504175 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:24.504282 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:24.504282 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:24.504463 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:24.504510 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:24.504511 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:24.505187 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:24.505188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:24.505724 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:24.505726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:24.506062 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:24.506062 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:24.506100 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:24.506564 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:24.506568 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:24.508385 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":79,"threads_started":1}
W20260926 20:49:24.508522 9528 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:24.504047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34675 (local address 127.0.98.62:36223)
0926 20:49:24.504110 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:24.504112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:24.504163 (+ 51us) server_negotiation.cc:407] Connection header received
0926 20:49:24.504227 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:24.504228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:24.504238 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:24.504277 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:24.504606 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:24.505105 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:24.505809 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:24.505993 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:24.506050 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:24.508458 (+ 2408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34675: BlockingRecv error: recv got EOF from 127.0.98.3:34675 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 26 20:49:25 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:49:25 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:49:25.265967 9531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:25.254663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.1:35797)
0926 20:49:25.254839 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:25.254855 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:25.254933 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:25.255119 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:25.255120 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:25.255281 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:25.255414 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:25.255417 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:25.256536 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:25.256537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:25.257353 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:25.257355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:25.257757 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:25.257758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:25.257815 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:25.264040 (+ 6225us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:25.264045 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:25.265867 (+ 1822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":75,"threads_started":1}
W20260926 20:49:25.266031 9532 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:25.254779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35797 (local address 127.0.98.62:36223)
0926 20:49:25.254950 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:25.254952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:25.254976 (+ 24us) server_negotiation.cc:407] Connection header received
0926 20:49:25.254999 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:25.255000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:25.255010 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:25.255061 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:25.255557 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:25.256467 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:25.257446 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:25.257660 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:25.257747 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:25.265957 (+ 8210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35797: BlockingRecv error: recv got EOF from 127.0.98.1:35797 (error 108)
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":109,"threads_started":1}
Sep 26 20:49:25 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:49:25 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:49:25.385713 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:25.381085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.2:51925)
0926 20:49:25.381193 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:25.381204 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:25.381266 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:25.381429 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:25.381430 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:25.381581 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:25.381629 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:25.381630 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:25.382602 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:25.382603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:25.383115 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:25.383117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:25.383465 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:25.383465 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:25.383504 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:25.383962 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:25.383966 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:25.385647 (+ 1681us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":63,"threads_started":1}
W20260926 20:49:25.385917 9532 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:25.381154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51925 (local address 127.0.98.62:36223)
0926 20:49:25.381204 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:25.381206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:25.381258 (+ 52us) server_negotiation.cc:407] Connection header received
0926 20:49:25.381330 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:25.381331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:25.381345 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:25.381419 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:25.381699 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:25.382515 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:25.383196 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:25.383402 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:25.383451 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:25.385770 (+ 2319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51925: BlockingRecv error: recv got EOF from 127.0.98.2:51925 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 26 20:49:25 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:49:25 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:49:25.513762 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:25.508977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.3:60795)
0926 20:49:25.509200 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:25.509216 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:25.509269 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:25.509478 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:25.509479 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:25.509644 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:25.509692 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:25.509694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:25.510341 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:25.510342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:25.510966 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:25.510969 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:25.511380 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:25.511380 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:25.511418 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:25.511922 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:25.511927 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:25.513691 (+ 1764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":176,"threads_started":1}
W20260926 20:49:25.513870 9532 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:25.509103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60795 (local address 127.0.98.62:36223)
0926 20:49:25.509204 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:25.509206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:25.509267 (+ 61us) server_negotiation.cc:407] Connection header received
0926 20:49:25.509323 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:25.509323 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:25.509398 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:25.509461 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:25.509756 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:25.510271 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:25.511068 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:25.511303 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:25.511371 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:25.513783 (+ 2412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60795: BlockingRecv error: recv got EOF from 127.0.98.3:60795 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 26 20:49:26 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:49:26 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:49:26.271325 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:26.266521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.1:38395)
0926 20:49:26.266693 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:26.266708 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:26.266778 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:26.266933 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:26.266934 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:26.267094 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:26.267143 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:26.267144 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:26.267897 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:26.267898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:26.268440 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:26.268441 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:26.268822 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:26.268823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:26.268861 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:26.269294 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:26.269298 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:26.271196 (+ 1898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":77,"threads_started":1}
W20260926 20:49:26.271371 9536 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:26.266648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38395 (local address 127.0.98.62:36223)
0926 20:49:26.266783 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:26.266785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:26.266794 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:49:26.266840 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:26.266840 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:26.266850 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:26.266910 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:26.267237 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:26.267817 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:26.268531 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:26.268751 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:26.268814 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:26.271294 (+ 2480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38395: BlockingRecv error: recv got EOF from 127.0.98.1:38395 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":92,"threads_started":1}
Sep 26 20:49:26 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:49:26 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:49:26.390661 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:26.386202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.2:38899)
0926 20:49:26.386315 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:26.386328 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:26.386387 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:26.386513 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:26.386514 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:26.386670 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:26.386717 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:26.386718 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:26.387375 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:26.387376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:26.387920 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:26.387922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:26.388301 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:26.388302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:26.388341 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:26.388853 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:26.388857 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:26.390602 (+ 1745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":56,"threads_started":1}
W20260926 20:49:26.390851 9536 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:26.386270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38899 (local address 127.0.98.62:36223)
0926 20:49:26.386366 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:26.386368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:26.386379 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:49:26.386448 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:26.386449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:26.386459 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:26.386508 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:26.386779 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:26.387297 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:26.388014 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:26.388214 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:26.388280 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:26.390763 (+ 2483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38899: BlockingRecv error: recv got EOF from 127.0.98.2:38899 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 26 20:49:26 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:49:26 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:49:26.519021 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:26.514284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.3:45893)
0926 20:49:26.514439 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:26.514451 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:26.514516 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:26.514638 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:26.514639 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:26.514814 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:26.514865 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:26.514867 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:26.515516 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:26.515517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:26.516081 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:26.516083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:26.516417 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:26.516418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:26.516461 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:26.517027 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:26.517049 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:26.518965 (+ 1916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":86,"threads_started":1}
W20260926 20:49:26.519148 9536 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:26.514392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45893 (local address 127.0.98.62:36223)
0926 20:49:26.514446 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:26.514448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:26.514521 (+ 73us) server_negotiation.cc:407] Connection header received
0926 20:49:26.514566 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:26.514567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:26.514578 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:26.514618 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:26.514930 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:26.515434 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:26.516157 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:26.516348 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:26.516399 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:26.519048 (+ 2649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45893: BlockingRecv error: recv got EOF from 127.0.98.3:45893 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 26 20:49:27 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:49:27 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:49:27.278111 9539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:27.271942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.1:55879)
0926 20:49:27.272120 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:27.272138 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:27.272198 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:27.272353 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:27.272353 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:27.272517 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:27.272563 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:27.272564 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:27.273505 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:27.273506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:27.274173 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:27.274176 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:27.274755 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:27.274756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:27.274833 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:27.275488 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:27.275493 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:27.278033 (+ 2540us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":88,"threads_started":1}
W20260926 20:49:27.278306 9540 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:27.272011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55879 (local address 127.0.98.62:36223)
0926 20:49:27.272166 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:27.272168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:27.272180 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:49:27.272262 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:27.272262 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:27.272273 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:27.272334 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:27.272640 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:27.273428 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:27.274292 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:27.274659 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:27.274739 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:27.278168 (+ 3429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55879: BlockingRecv error: recv got EOF from 127.0.98.1:55879 (error 108)
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":68,"threads_started":1}
Sep 26 20:49:27 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:49:27 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:49:27.396143 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:27.391212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.2:46219)
0926 20:49:27.391353 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:27.391366 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:27.391439 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:27.391588 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:27.391589 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:27.391843 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:27.391903 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:27.391905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:27.392607 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:27.392609 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:27.393290 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:27.393292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:27.393609 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:27.393610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:27.393652 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:27.394098 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:27.394103 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:27.396083 (+ 1980us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":85,"threads_started":1}
W20260926 20:49:27.396312 9540 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:27.391322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46219 (local address 127.0.98.62:36223)
0926 20:49:27.391394 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:27.391396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:27.391434 (+ 38us) server_negotiation.cc:407] Connection header received
0926 20:49:27.391499 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:27.391500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:27.391511 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:27.391565 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:27.391977 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:27.392468 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:27.393361 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:27.393545 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:27.393598 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:27.396215 (+ 2617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46219: BlockingRecv error: recv got EOF from 127.0.98.2:46219 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 26 20:49:27 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:49:27 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:49:27.524457 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:27.519564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.3:48237)
0926 20:49:27.519776 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:27.519786 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:27.519848 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:27.519980 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:27.519981 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:27.520141 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:27.520189 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:27.520191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:27.520843 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:27.520844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:27.521404 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:27.521406 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:27.521725 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:27.521726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:27.521765 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:27.522306 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:27.522310 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:27.524399 (+ 2089us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":122,"threads_started":1}
W20260926 20:49:27.524561 9540 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:27.519662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48237 (local address 127.0.98.62:36223)
0926 20:49:27.519732 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:27.519739 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:27.519859 (+ 120us) server_negotiation.cc:407] Connection header received
0926 20:49:27.519901 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:27.519902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:27.519914 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:27.519956 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:27.520251 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:27.520731 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:27.521483 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:27.521660 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:27.521710 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:27.524492 (+ 2782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48237: BlockingRecv error: recv got EOF from 127.0.98.3:48237 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 26 20:49:28 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:49:28 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:49:28.283334 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:28.278630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.1:36733)
0926 20:49:28.278800 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:28.278811 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:28.278862 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:28.279017 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:28.279018 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:28.279180 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:28.279229 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:28.279231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:28.279937 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:28.279938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:28.280478 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:28.280481 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:28.280839 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:28.280840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:28.280881 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:28.281411 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:28.281416 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:28.283275 (+ 1859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":101,"threads_started":1}
W20260926 20:49:28.283460 9544 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:28.278748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36733 (local address 127.0.98.62:36223)
0926 20:49:28.278889 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:28.278891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:28.278901 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:49:28.278931 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:28.278932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:28.278942 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:28.279001 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:28.279290 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:28.279870 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:28.280572 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:28.280769 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:28.280833 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:28.283373 (+ 2540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36733: BlockingRecv error: recv got EOF from 127.0.98.1:36733 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":95,"threads_started":1}
Sep 26 20:49:28 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:49:28 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:49:28.401666 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:28.396760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.2:41361)
0926 20:49:28.396924 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:28.396938 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:28.397042 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:28.397238 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:28.397239 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:28.397473 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:28.397534 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:28.397539 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:28.398191 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:28.398192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:28.398752 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:28.398753 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:28.399087 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:28.399088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:28.399124 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:28.399636 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:28.399641 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:28.401589 (+ 1948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":86,"threads_started":1}
W20260926 20:49:28.401811 9544 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:28.396890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41361 (local address 127.0.98.62:36223)
0926 20:49:28.396960 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:28.396962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:28.397042 (+ 80us) server_negotiation.cc:407] Connection header received
0926 20:49:28.397113 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:28.397114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:28.397127 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:28.397188 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:28.397616 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:28.398122 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:28.398826 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:28.399022 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:28.399077 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:28.401693 (+ 2616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41361: BlockingRecv error: recv got EOF from 127.0.98.2:41361 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 26 20:49:28 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:49:28 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:49:28.529385 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:28.524941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.3:53455)
0926 20:49:28.525109 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:28.525127 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:28.525183 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:28.525300 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:28.525300 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:28.525452 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:28.525499 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:28.525501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:28.526132 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:28.526132 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:28.526655 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:28.526656 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:28.527030 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:28.527032 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:28.527099 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:28.527625 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:28.527630 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:28.529317 (+ 1687us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":126,"threads_started":1}
W20260926 20:49:28.529449 9544 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:28.525027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53455 (local address 127.0.98.62:36223)
0926 20:49:28.525100 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:28.525102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:28.525182 (+ 80us) server_negotiation.cc:407] Connection header received
0926 20:49:28.525227 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:28.525228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:28.525240 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:28.525288 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:28.525553 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:28.526062 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:28.526757 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:28.526939 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:28.527009 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:28.529402 (+ 2393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53455: BlockingRecv error: recv got EOF from 127.0.98.3:53455 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 26 20:49:29 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:49:29 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:49:29.295269 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:29.283874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.1:55483)
0926 20:49:29.283990 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:29.284002 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:29.284064 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:29.284273 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:29.284275 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:29.284500 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:29.284569 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:29.284572 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:29.285493 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:29.285494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:29.286167 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:29.286169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:29.286552 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:29.286553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:29.286614 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:29.293124 (+ 6510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:29.293129 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:29.295158 (+ 2029us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":69,"threads_started":1}
W20260926 20:49:29.295331 9548 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:29.283923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55483 (local address 127.0.98.62:36223)
0926 20:49:29.284038 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:29.284040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:29.284049 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:49:29.284148 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:29.284149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:29.284162 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:29.284249 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:29.284640 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:29.285406 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:29.286262 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:29.286468 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:29.286550 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:29.295255 (+ 8705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55483: BlockingRecv error: recv got EOF from 127.0.98.1:55483 (error 108)
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":64,"threads_started":1}
Sep 26 20:49:29 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:49:29 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:49:29.406942 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:29.402134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.2:37847)
0926 20:49:29.402355 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:29.402367 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:29.402460 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:29.402613 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:29.402614 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:29.402791 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:29.402838 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:29.402839 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:29.403516 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:29.403517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:29.404111 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:29.404113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:29.404522 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:29.404523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:29.404589 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:29.405053 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:29.405057 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:29.406882 (+ 1825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":104,"threads_started":1}
W20260926 20:49:29.407086 9548 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:29.402246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37847 (local address 127.0.98.62:36223)
0926 20:49:29.402350 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:29.402352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:29.402441 (+ 89us) server_negotiation.cc:407] Connection header received
0926 20:49:29.402520 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:29.402521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:29.402540 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:29.402605 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:29.402929 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:29.403429 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:29.404225 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:29.404417 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:29.404508 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:29.407002 (+ 2494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37847: BlockingRecv error: recv got EOF from 127.0.98.2:37847 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 26 20:49:29 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:49:29 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:49:29.534360 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:29.529862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.3:43769)
0926 20:49:29.529996 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:29.530006 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:29.530060 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:29.530191 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:29.530210 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:29.530367 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:29.530413 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:29.530415 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:29.531065 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:29.531066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:29.531613 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:29.531615 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:29.531994 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:29.531995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:29.532046 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:29.532503 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:29.532507 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:29.534302 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":91,"threads_started":1}
W20260926 20:49:29.534616 9548 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:29.529914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43769 (local address 127.0.98.62:36223)
0926 20:49:29.529961 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:29.529964 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:29.530050 (+ 86us) server_negotiation.cc:407] Connection header received
0926 20:49:29.530114 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:29.530114 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:29.530126 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:29.530163 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:29.530476 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:29.530973 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:29.531697 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:29.531924 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:29.531977 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:29.534569 (+ 2592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43769: BlockingRecv error: recv got EOF from 127.0.98.3:43769 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 26 20:49:30 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:49:30 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:49:30.300793 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:30.295821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.1:46671)
0926 20:49:30.295979 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:30.295991 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:30.296041 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:30.296292 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:30.296293 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:30.296454 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:30.296503 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:30.296504 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:30.297164 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:30.297165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:30.297708 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:30.297710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:30.298089 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:30.298090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:30.298178 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:30.298629 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:30.298634 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:30.300734 (+ 2100us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":94,"threads_started":1}
W20260926 20:49:30.300948 9552 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:30.296008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46671 (local address 127.0.98.62:36223)
0926 20:49:30.296154 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:30.296156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:30.296164 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:49:30.296192 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:30.296192 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:30.296202 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:30.296267 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:30.296591 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:30.297085 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:30.297797 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:30.297993 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:30.298049 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:30.300827 (+ 2778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46671: BlockingRecv error: recv got EOF from 127.0.98.1:46671 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":83,"threads_started":1}
Sep 26 20:49:30 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:49:30 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:49:30.412938 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:30.407467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.2:44911)
0926 20:49:30.407813 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:30.407824 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:30.407894 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:30.408152 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:30.408154 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:30.408371 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:30.408435 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:30.408438 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:30.409202 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:30.409203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:30.409774 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:30.409776 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:30.410317 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:30.410318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:30.410393 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:30.410905 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:30.410910 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:30.412879 (+ 1969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":234,"threads_started":1}
W20260926 20:49:30.413146 9552 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:30.407526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44911 (local address 127.0.98.62:36223)
0926 20:49:30.407735 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:30.407737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:30.407908 (+ 171us) server_negotiation.cc:407] Connection header received
0926 20:49:30.407960 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:30.407961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:30.407982 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:30.408024 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:30.408522 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:30.409094 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:30.409923 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:30.410193 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:30.410252 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:30.413022 (+ 2770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44911: BlockingRecv error: recv got EOF from 127.0.98.2:44911 (error 108)
Metrics: {"server-negotiator.queue_time_us":181}
Sep 26 20:49:30 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:49:30 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:49:30.539811 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:30.534918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.3:50955)
0926 20:49:30.535072 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:30.535090 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:30.535146 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:30.535276 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:30.535277 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:30.535444 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:30.535494 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:30.535496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:30.536194 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:30.536195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:30.536736 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:30.536738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:30.537093 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:30.537094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:30.537144 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:30.537643 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:30.537648 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:30.539702 (+ 2054us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":106,"threads_started":1}
W20260926 20:49:30.539872 9552 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:30.534970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50955 (local address 127.0.98.62:36223)
0926 20:49:30.535042 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:30.535043 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:30.535148 (+ 105us) server_negotiation.cc:407] Connection header received
0926 20:49:30.535190 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:30.535191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:30.535203 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:30.535257 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:30.535557 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:30.536121 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:30.536819 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:30.536999 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:30.537056 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:30.539825 (+ 2769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50955: BlockingRecv error: recv got EOF from 127.0.98.3:50955 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 26 20:49:31 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:49:31 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:49:31.306113 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:31.301288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.1:50293)
0926 20:49:31.301431 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:31.301445 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:31.301508 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:31.301707 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:31.301707 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:31.301871 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:31.301917 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:31.301918 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:31.302599 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:31.302600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:31.303198 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:31.303200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:31.303536 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:31.303537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:31.303586 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:31.304063 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:31.304068 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:31.305984 (+ 1916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":89,"threads_started":1}
W20260926 20:49:31.306154 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:31.301439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50293 (local address 127.0.98.62:36223)
0926 20:49:31.301567 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:31.301569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:31.301577 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:49:31.301599 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:31.301599 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:31.301610 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:31.301687 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:31.301993 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:31.302508 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:31.303277 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:31.303467 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:31.303517 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:31.306079 (+ 2562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50293: BlockingRecv error: recv got EOF from 127.0.98.1:50293 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":69,"threads_started":1}
Sep 26 20:49:31 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:49:31 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:49:31.418697 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:31.413501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.2:60617)
0926 20:49:31.413725 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:31.413737 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:31.413840 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:31.413996 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:31.413998 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:31.414189 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:31.414236 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:31.414238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:31.415053 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:31.415055 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:31.415614 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:31.415615 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:31.416051 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:31.416052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:31.416115 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:31.416636 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:31.416641 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:31.418609 (+ 1968us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":122,"threads_started":1}
W20260926 20:49:31.418869 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:31.413599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60617 (local address 127.0.98.62:36223)
0926 20:49:31.413711 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:31.413713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:31.413820 (+ 107us) server_negotiation.cc:407] Connection header received
0926 20:49:31.413900 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:31.413901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:31.413915 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:31.413989 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:31.414324 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:31.414960 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:31.415708 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:31.415963 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:31.416048 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:31.418774 (+ 2726us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60617: BlockingRecv error: recv got EOF from 127.0.98.2:60617 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 26 20:49:31 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:49:31 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:49:31.545037 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:31.540348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.3:47037)
0926 20:49:31.540526 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:31.540539 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:31.540614 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:31.540800 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:31.540801 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:31.540987 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:31.541060 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:31.541062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:31.541717 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:31.541718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:31.542260 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:31.542262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:31.542582 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:31.542583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:31.542658 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:31.543142 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:31.543146 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:31.544954 (+ 1808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":72,"threads_started":1}
W20260926 20:49:31.545258 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:31.540471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47037 (local address 127.0.98.62:36223)
0926 20:49:31.540541 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:31.540543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:31.540612 (+ 69us) server_negotiation.cc:407] Connection header received
0926 20:49:31.540661 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:31.540662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:31.540674 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:31.540722 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:31.541140 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:31.541633 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:31.542335 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:31.542514 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:31.542564 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:31.545157 (+ 2593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47037: BlockingRecv error: recv got EOF from 127.0.98.3:47037 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 26 20:49:32 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:49:32 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:49:32.311405 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:32.306612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.1:33037)
0926 20:49:32.306745 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:32.306756 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:32.306814 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:32.306952 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:32.306953 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:32.307113 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:32.307163 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:32.307165 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:32.307873 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:32.307874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:32.308399 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:32.308401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:32.308742 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:32.308743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:32.308783 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:32.309286 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:32.309290 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:32.311345 (+ 2055us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":80,"threads_started":1}
W20260926 20:49:32.311550 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:32.306670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33037 (local address 127.0.98.62:36223)
0926 20:49:32.306794 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:32.306798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:32.306812 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:49:32.306867 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:32.306867 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:32.306877 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:32.306935 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:32.307263 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:32.307800 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:32.308470 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:32.308679 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:32.308731 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:32.311457 (+ 2726us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33037: BlockingRecv error: recv got EOF from 127.0.98.1:33037 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":75,"threads_started":1}
Sep 26 20:49:32 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:49:32 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:49:32.423794 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:32.419246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.2:39323)
0926 20:49:32.419393 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:32.419411 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:32.419468 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:32.419594 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:32.419595 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:32.419778 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:32.419827 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:32.419829 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:32.420431 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:32.420432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:32.420945 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:32.420946 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:32.421278 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:32.421279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:32.421318 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:32.421842 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:32.421846 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:32.423681 (+ 1835us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":94,"threads_started":1}
W20260926 20:49:32.423871 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:32.419328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39323 (local address 127.0.98.62:36223)
0926 20:49:32.419402 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:32.419403 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:32.419465 (+ 62us) server_negotiation.cc:407] Connection header received
0926 20:49:32.419510 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:32.419510 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:32.419522 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:32.419573 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:32.419878 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:32.420365 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:32.421024 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:32.421212 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:32.421267 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:32.423823 (+ 2556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39323: BlockingRecv error: recv got EOF from 127.0.98.2:39323 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 26 20:49:32 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:49:32 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:49:32.550325 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:32.545658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.3:52977)
0926 20:49:32.545826 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:32.545839 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:32.545914 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:32.546094 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:32.546096 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:32.546317 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:32.546375 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:32.546377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:32.547015 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:32.547016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:32.547561 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:32.547562 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:32.547914 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:32.547915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:32.547963 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:32.548400 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:32.548404 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:32.550266 (+ 1862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":95,"threads_started":1}
W20260926 20:49:32.550443 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:32.545761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52977 (local address 127.0.98.62:36223)
0926 20:49:32.545815 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:32.545817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:32.545920 (+ 103us) server_negotiation.cc:407] Connection header received
0926 20:49:32.545996 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:32.545998 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:32.546013 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:32.546064 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:32.546440 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:32.546926 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:32.547632 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:32.547839 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:32.547889 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:32.550366 (+ 2477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52977: BlockingRecv error: recv got EOF from 127.0.98.3:52977 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 26 20:49:33 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:49:33 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:49:33.316661 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:33.311873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.1:54825)
0926 20:49:33.312016 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:33.312029 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:33.312108 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:33.312251 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:33.312252 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:33.312413 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:33.312458 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:33.312459 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:33.313433 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:33.313434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:33.313973 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:33.313975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:33.314319 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:33.314319 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:33.314358 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:33.314796 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:33.314800 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:33.316600 (+ 1800us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":68,"threads_started":1}
W20260926 20:49:33.316816 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:33.311967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54825 (local address 127.0.98.62:36223)
0926 20:49:33.312108 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:33.312109 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:33.312119 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:49:33.312159 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:33.312159 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:33.312170 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:33.312230 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:33.312613 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:33.313365 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:33.314050 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:33.314253 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:33.314307 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:33.316712 (+ 2405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54825: BlockingRecv error: recv got EOF from 127.0.98.1:54825 (error 108)
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":64,"threads_started":1}
Sep 26 20:49:33 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:49:33 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:49:33.429139 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:33.424347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.2:37305)
0926 20:49:33.424519 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:33.424567 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:33.424642 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:33.424807 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:33.424808 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:33.425042 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:33.425091 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:33.425093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:33.425770 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:33.425771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:33.426305 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:33.426307 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:33.426680 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:33.426681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:33.426741 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:33.427249 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:33.427254 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:33.429051 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":93,"threads_started":1}
W20260926 20:49:33.429219 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:33.424498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37305 (local address 127.0.98.62:36223)
0926 20:49:33.424594 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:33.424596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:33.424621 (+ 25us) server_negotiation.cc:407] Connection header received
0926 20:49:33.424714 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:33.424715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:33.424726 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:33.424803 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:33.425179 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:33.425684 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:33.426390 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:33.426593 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:33.426679 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:33.429157 (+ 2478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37305: BlockingRecv error: recv got EOF from 127.0.98.2:37305 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 26 20:49:33 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:49:33 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:49:33.556128 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:33.550914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.3:40587)
0926 20:49:33.551160 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:33.551174 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:33.551254 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:33.551423 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:33.551424 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:33.551609 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:33.551656 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:33.551657 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:33.552371 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:33.552372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:33.553085 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:33.553087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:33.553521 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:33.553522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:33.553582 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:33.554045 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:33.554049 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:33.556009 (+ 1960us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":138,"threads_started":1}
W20260926 20:49:33.556164 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:33.551103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40587 (local address 127.0.98.62:36223)
0926 20:49:33.551198 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:33.551200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:33.551234 (+ 34us) server_negotiation.cc:407] Connection header received
0926 20:49:33.551329 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:33.551330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:33.551349 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:33.551417 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:33.551771 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:33.552282 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:33.553173 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:33.553420 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:33.553519 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:33.556095 (+ 2576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40587: BlockingRecv error: recv got EOF from 127.0.98.3:40587 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 26 20:49:34 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:49:34 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:49:34.325498 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:34.317212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.1:33067)
0926 20:49:34.317376 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:34.317390 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:34.317455 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:34.317634 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:34.317635 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:34.317797 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:34.317847 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:34.317848 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:34.318564 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:34.318565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:34.319075 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:34.319076 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:34.319501 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:34.319502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:34.319564 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:34.323489 (+ 3925us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:34.323495 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:34.325437 (+ 1942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":72,"threads_started":1}
W20260926 20:49:34.325696 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:34.317269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33067 (local address 127.0.98.62:36223)
0926 20:49:34.317458 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:34.317460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:34.317476 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:49:34.317505 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:34.317506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:34.317517 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:34.317597 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:34.317919 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:34.318494 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:34.319208 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:34.319414 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:34.319498 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:34.325606 (+ 6108us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33067: BlockingRecv error: recv got EOF from 127.0.98.1:33067 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":92,"threads_started":1}
Sep 26 20:49:34 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:49:34 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:49:34.434500 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:34.429638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.2:46223)
0926 20:49:34.429774 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:34.429785 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:34.429845 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:34.429979 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:34.429980 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:34.430153 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:34.430203 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:34.430205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:34.430818 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:34.430819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:34.431415 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:34.431418 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:34.431900 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:34.431901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:34.431950 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:34.432474 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:34.432479 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:34.434437 (+ 1958us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":83,"threads_started":1}
W20260926 20:49:34.434621 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:34.429702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46223 (local address 127.0.98.62:36223)
0926 20:49:34.429767 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:34.429769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:34.429829 (+ 60us) server_negotiation.cc:407] Connection header received
0926 20:49:34.429899 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:34.429900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:34.429909 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:34.429960 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:34.430264 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:34.430749 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:34.431530 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:34.431718 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:34.431884 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:34.434534 (+ 2650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46223: BlockingRecv error: recv got EOF from 127.0.98.2:46223 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 26 20:49:34 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:49:34 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:49:34.561471 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:34.556684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.3:38915)
0926 20:49:34.556818 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:34.556829 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:34.556885 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:34.557016 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:34.557016 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:34.557176 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:34.557233 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:34.557235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:34.557887 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:34.557889 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:34.558445 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:34.558447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:34.558766 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:34.558767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:34.558810 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:34.559345 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:34.559349 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:34.561401 (+ 2052us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
W20260926 20:49:34.561560 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:34.556821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38915 (local address 127.0.98.62:36223)
0926 20:49:34.556878 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:34.556880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:34.556889 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:49:34.556930 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:34.556930 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:34.556941 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:34.556998 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:34.557303 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:34.557795 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:34.558526 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:34.558702 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:34.558751 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:34.561493 (+ 2742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38915: BlockingRecv error: recv got EOF from 127.0.98.3:38915 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 26 20:49:35 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:49:35 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:49:35.331211 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:35.326005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.1:33975)
0926 20:49:35.326205 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:35.326216 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:35.326281 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:35.326419 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:35.326419 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:35.326590 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:35.326637 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:35.326639 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:35.327286 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:35.327287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:35.327874 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:35.327876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:35.328252 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:35.328253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:35.328291 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:35.328737 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:35.328742 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:35.331108 (+ 2366us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":130,"threads_started":1}
W20260926 20:49:35.331267 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:35.326113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33975 (local address 127.0.98.62:36223)
0926 20:49:35.326216 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:35.326218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:35.326268 (+ 50us) server_negotiation.cc:407] Connection header received
0926 20:49:35.326337 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:35.326338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:35.326348 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:35.326397 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:35.326705 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:35.327211 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:35.327979 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:35.328187 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:35.328242 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:35.331188 (+ 2946us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33975: BlockingRecv error: recv got EOF from 127.0.98.1:33975 (error 108)
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":48,"threads_started":1}
Sep 26 20:49:35 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:49:35 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:49:35.439489 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:35.434950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.2:37595)
0926 20:49:35.435083 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:35.435093 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:35.435157 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:35.435266 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:35.435266 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:35.435421 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:35.435470 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:35.435471 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:35.436198 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:35.436199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:35.436739 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:35.436741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:35.437049 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:35.437050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:35.437100 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:35.437532 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:35.437536 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:35.439368 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
W20260926 20:49:35.439550 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:35.435070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37595 (local address 127.0.98.62:36223)
0926 20:49:35.435112 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:35.435114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:35.435142 (+ 28us) server_negotiation.cc:407] Connection header received
0926 20:49:35.435204 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:35.435204 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:35.435214 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:35.435261 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:35.435557 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:35.436121 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:35.436807 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:35.436984 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:35.437032 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:35.439474 (+ 2442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37595: BlockingRecv error: recv got EOF from 127.0.98.2:37595 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Sep 26 20:49:35 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:49:35 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:49:35.566448 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:35.561945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.3:32865)
0926 20:49:35.562103 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:35.562112 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:35.562166 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:35.562300 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:35.562301 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:35.562501 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:35.562548 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:35.562550 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:35.563174 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:35.563174 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:35.563702 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:35.563703 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:35.564089 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:35.564090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:35.564137 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:35.564576 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:35.564581 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:35.566341 (+ 1760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":69,"threads_started":1}
W20260926 20:49:35.566483 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:35.562004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32865 (local address 127.0.98.62:36223)
0926 20:49:35.562069 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:35.562071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:35.562169 (+ 98us) server_negotiation.cc:407] Connection header received
0926 20:49:35.562217 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:35.562218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:35.562230 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:35.562288 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:35.562617 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:35.563106 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:35.563835 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:35.564009 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:35.564061 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:35.566417 (+ 2356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:32865: BlockingRecv error: recv got EOF from 127.0.98.3:32865 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 26 20:49:36 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:49:36 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:49:36.336575 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:36.331836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.1:58353)
0926 20:49:36.331986 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:36.332000 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:36.332074 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:36.332268 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:36.332269 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:36.332443 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:36.332487 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:36.332489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:36.333230 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:36.333231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:36.333778 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:36.333779 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:36.334171 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:36.334172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:36.334226 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:36.334661 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:36.334666 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:36.336496 (+ 1830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":79,"threads_started":1}
W20260926 20:49:36.336723 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:36.331862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58353 (local address 127.0.98.62:36223)
0926 20:49:36.331991 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:36.331993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:36.332053 (+ 60us) server_negotiation.cc:407] Connection header received
0926 20:49:36.332159 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:36.332159 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:36.332174 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:36.332250 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:36.332572 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:36.333149 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:36.333863 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:36.334085 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:36.334170 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:36.336625 (+ 2455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58353: BlockingRecv error: recv got EOF from 127.0.98.1:58353 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":53,"threads_started":1}
Sep 26 20:49:36 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:49:36 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:49:36.444661 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:36.440124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.2:46885)
0926 20:49:36.440256 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:36.440268 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:36.440331 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:36.440478 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:36.440479 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:36.440648 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:36.440694 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:36.440695 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:36.441325 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:36.441325 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:36.441834 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:36.441835 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:36.442214 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:36.442215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:36.442261 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:36.442709 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:36.442714 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:36.444601 (+ 1887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":80,"threads_started":1}
W20260926 20:49:36.444790 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:36.440282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46885 (local address 127.0.98.62:36223)
0926 20:49:36.440347 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:36.440349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:36.440358 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:49:36.440392 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:36.440393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:36.440403 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:36.440457 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:36.440762 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:36.441257 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:36.441938 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:36.442125 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:36.442179 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:36.444706 (+ 2527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46885: BlockingRecv error: recv got EOF from 127.0.98.2:46885 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 26 20:49:36 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:49:36 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:49:36.571905 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:36.566941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.3:39731)
0926 20:49:36.567085 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:36.567099 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:36.567161 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:36.567301 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:36.567302 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:36.567476 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:36.567524 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:36.567525 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:36.568396 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:36.568397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:36.568935 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:36.568937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:36.569263 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:36.569263 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:36.569312 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:36.569843 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:36.569847 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:36.571799 (+ 1952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":98,"threads_started":1}
W20260926 20:49:36.571934 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:36.567015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39731 (local address 127.0.98.62:36223)
0926 20:49:36.567086 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:36.567087 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:36.567161 (+ 74us) server_negotiation.cc:407] Connection header received
0926 20:49:36.567220 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:36.567221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:36.567232 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:36.567274 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:36.567601 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:36.568299 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:36.569017 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:36.569196 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:36.569250 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:36.571868 (+ 2618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39731: BlockingRecv error: recv got EOF from 127.0.98.3:39731 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 26 20:49:37 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:49:37 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:49:37.341918 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:37.337057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.1:55271)
0926 20:49:37.337218 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:37.337231 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:37.337293 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:37.337440 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:37.337440 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:37.337597 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:37.337650 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:37.337651 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:37.338283 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:37.338284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:37.338802 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:37.338804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:37.339164 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:37.339166 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:37.339215 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:37.339840 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:37.339845 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:37.341858 (+ 2013us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":71,"threads_started":1}
W20260926 20:49:37.342087 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:37.337117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55271 (local address 127.0.98.62:36223)
0926 20:49:37.337268 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:37.337270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:37.337280 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:49:37.337353 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:37.337353 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:37.337363 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:37.337420 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:37.337715 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:37.338211 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:37.338876 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:37.339079 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:37.339130 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:37.342003 (+ 2873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55271: BlockingRecv error: recv got EOF from 127.0.98.1:55271 (error 108)
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":90,"threads_started":1}
Sep 26 20:49:37 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:49:37 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:49:37.449681 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:37.445252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.2:43161)
0926 20:49:37.445410 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:37.445428 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:37.445486 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:37.445604 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:37.445605 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:37.445762 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:37.445810 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:37.445811 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:37.446448 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:37.446449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:37.446962 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:37.446963 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:37.447286 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:37.447286 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:37.447325 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:37.447780 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:37.447785 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:37.449570 (+ 1785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":108,"threads_started":1}
W20260926 20:49:37.449720 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:37.445313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43161 (local address 127.0.98.62:36223)
0926 20:49:37.445388 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:37.445390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:37.445489 (+ 99us) server_negotiation.cc:407] Connection header received
0926 20:49:37.445536 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:37.445537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:37.445547 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:37.445600 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:37.445867 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:37.446377 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:37.447053 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:37.447227 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:37.447272 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:37.449658 (+ 2386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43161: BlockingRecv error: recv got EOF from 127.0.98.2:43161 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 26 20:49:37 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:49:37 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:49:37.577147 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:37.572420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.3:46367)
0926 20:49:37.572596 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:37.572611 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:37.572704 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:37.572892 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:37.572893 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:37.573082 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:37.573129 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:37.573131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:37.573809 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:37.573810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:37.574362 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:37.574363 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:37.574737 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:37.574738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:37.574795 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:37.575294 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:37.575299 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:37.577050 (+ 1751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":85,"threads_started":1}
W20260926 20:49:37.577190 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:37.572527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46367 (local address 127.0.98.62:36223)
0926 20:49:37.572600 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:37.572602 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:37.572685 (+ 83us) server_negotiation.cc:407] Connection header received
0926 20:49:37.572795 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:37.572796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:37.572819 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:37.572886 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:37.573212 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:37.573718 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:37.574462 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:37.574652 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:37.574736 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:37.577126 (+ 2390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46367: BlockingRecv error: recv got EOF from 127.0.98.3:46367 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 26 20:49:38 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:49:38 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:49:38.347102 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:38.342410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.1:53731)
0926 20:49:38.342552 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:38.342571 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:38.342615 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:38.342792 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:38.342793 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:38.342949 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:38.343023 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:38.343025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:38.343687 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:38.343688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:38.344285 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:38.344287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:38.344617 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:38.344618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:38.344659 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:38.345080 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:38.345084 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:38.347033 (+ 1949us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":109,"threads_started":1}
W20260926 20:49:38.347231 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:38.342541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53731 (local address 127.0.98.62:36223)
0926 20:49:38.342675 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:38.342677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:38.342684 (+ 7us) server_negotiation.cc:407] Connection header received
0926 20:49:38.342705 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:38.342705 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:38.342715 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:38.342777 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:38.343095 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:38.343611 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:38.344352 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:38.344549 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:38.344598 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:38.347142 (+ 2544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53731: BlockingRecv error: recv got EOF from 127.0.98.1:53731 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":80,"threads_started":1}
Sep 26 20:49:38 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:49:38 dist-test-slave-hwlj krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:49:38.455188 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:38.450227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36223 (local address 127.0.98.2:54389)
0926 20:49:38.450361 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:38.450374 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:38.450428 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:38.450588 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:38.450589 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:38.450748 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:38.450798 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:38.450800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:38.451512 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:38.451513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:38.452066 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:38.452068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:38.452452 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:38.452454 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:38.452510 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:38.453065 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:38.453070 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:38.455070 (+ 2000us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36223: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":79,"threads_started":1}
W20260926 20:49:38.455233 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:38.450414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54389 (local address 127.0.98.62:36223)
0926 20:49:38.450469 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:38.450471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:38.450479 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:49:38.450504 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:38.450505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:38.450516 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:38.450572 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:38.450865 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:38.451423 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:38.452155 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:38.452374 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:38.452440 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:38.455164 (+ 2724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54389: BlockingRecv error: recv got EOF from 127.0.98.2:54389 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 20:49:38.501904 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 9109
I20260926 20:49:38.506379 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 9261
I20260926 20:49:38.510859 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 9394
I20260926 20:49:38.515247 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 9039
2026-09-26T20:49:38Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17434 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T20:49:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:49:38Z Disabled control of system clock
I20260926 20:49:38.542631 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43561
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43561
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 20:49:38.622254 9589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:38.622505 9589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:38.622539 9589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:38.623729 9589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:49:38.623828 9589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:38.623849 9589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:38.623870 9589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:49:38.623901 9589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:49:38.625267 9589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43561
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:38.625537 9589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:38.625823 9589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:38.630839 9594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:38.630966 9589 server_base.cc:1034] running on GCE node
W20260926 20:49:38.630869 9597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:38.630875 9595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:38.631402 9589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:38.631728 9589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:38.632966 9589 hybrid_clock.cc:648] HybridClock initialized: now 1790455778632951 us; error 30 us; skew 500 ppm
I20260926 20:49:38.634234 9589 webserver.cc:460] Webserver started at http://127.0.98.62:45061/ using document root <none> and password file <none>
I20260926 20:49:38.634438 9589 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:38.634480 9589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:38.634568 9589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:49:38.635658 9589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/master-0/data/instance:
uuid: "a6fe423fcbe142609ed354162fcf29a7"
format_stamp: "Formatted at 2026-09-26 20:49:38 on dist-test-slave-hwlj"
server_key: "e171bc9fc0603dcdb535e609783447a6"
server_key_iv: "10b4f871d09382014e7225b7d78d2a68"
server_key_version: "encryptionkey@0"
I20260926 20:49:38.636076 9589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/master-0/wal/instance:
uuid: "a6fe423fcbe142609ed354162fcf29a7"
format_stamp: "Formatted at 2026-09-26 20:49:38 on dist-test-slave-hwlj"
server_key: "e171bc9fc0603dcdb535e609783447a6"
server_key_iv: "10b4f871d09382014e7225b7d78d2a68"
server_key_version: "encryptionkey@0"
I20260926 20:49:38.637326 9589 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:49:38.638024 9603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:38.638191 9589 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:49:38.638299 9589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "a6fe423fcbe142609ed354162fcf29a7"
format_stamp: "Formatted at 2026-09-26 20:49:38 on dist-test-slave-hwlj"
server_key: "e171bc9fc0603dcdb535e609783447a6"
server_key_iv: "10b4f871d09382014e7225b7d78d2a68"
server_key_version: "encryptionkey@0"
I20260926 20:49:38.638375 9589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:38.651849 9589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:38.652184 9589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:38.652336 9589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:38.656476 9589 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43561
I20260926 20:49:38.656522 9655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43561 every 8 connection(s)
I20260926 20:49:38.656881 9589 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:49:38.657529 9656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:49:38.660043 9656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6fe423fcbe142609ed354162fcf29a7: Bootstrap starting.
I20260926 20:49:38.660568 9656 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6fe423fcbe142609ed354162fcf29a7: Neither blocks nor log segments found. Creating new log.
I20260926 20:49:38.660795 9656 log.cc:824] T 00000000000000000000000000000000 P a6fe423fcbe142609ed354162fcf29a7: Log is configured to *not* fsync() on all Append() calls
I20260926 20:49:38.661418 9656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6fe423fcbe142609ed354162fcf29a7: No bootstrap required, opened a new log
I20260926 20:49:38.662911 9656 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6fe423fcbe142609ed354162fcf29a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6fe423fcbe142609ed354162fcf29a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43561 } }
I20260926 20:49:38.663050 9656 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6fe423fcbe142609ed354162fcf29a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:49:38.663072 9656 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6fe423fcbe142609ed354162fcf29a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6fe423fcbe142609ed354162fcf29a7, State: Initialized, Role: FOLLOWER
I20260926 20:49:38.663213 9656 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6fe423fcbe142609ed354162fcf29a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6fe423fcbe142609ed354162fcf29a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43561 } }
I20260926 20:49:38.663296 9656 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6fe423fcbe142609ed354162fcf29a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:49:38.663319 9656 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6fe423fcbe142609ed354162fcf29a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:49:38.663347 9656 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6fe423fcbe142609ed354162fcf29a7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:49:38.663934 9656 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6fe423fcbe142609ed354162fcf29a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6fe423fcbe142609ed354162fcf29a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43561 } }
I20260926 20:49:38.664029 9656 leader_election.cc:302] T 00000000000000000000000000000000 P a6fe423fcbe142609ed354162fcf29a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6fe423fcbe142609ed354162fcf29a7; no voters:
I20260926 20:49:38.664179 9656 leader_election.cc:288] T 00000000000000000000000000000000 P a6fe423fcbe142609ed354162fcf29a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:49:38.664206 9659 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6fe423fcbe142609ed354162fcf29a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:49:38.664311 9659 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6fe423fcbe142609ed354162fcf29a7 [term 1 LEADER]: Becoming Leader. State: Replica: a6fe423fcbe142609ed354162fcf29a7, State: Running, Role: LEADER
I20260926 20:49:38.664400 9659 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6fe423fcbe142609ed354162fcf29a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6fe423fcbe142609ed354162fcf29a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43561 } }
I20260926 20:49:38.664436 9656 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6fe423fcbe142609ed354162fcf29a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:49:38.664827 9660 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6fe423fcbe142609ed354162fcf29a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6fe423fcbe142609ed354162fcf29a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6fe423fcbe142609ed354162fcf29a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43561 } } }
I20260926 20:49:38.664916 9660 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6fe423fcbe142609ed354162fcf29a7 [sys.catalog]: This master's current role is: LEADER
I20260926 20:49:38.665414 9661 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6fe423fcbe142609ed354162fcf29a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6fe423fcbe142609ed354162fcf29a7. Latest consensus state: current_term: 1 leader_uuid: "a6fe423fcbe142609ed354162fcf29a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6fe423fcbe142609ed354162fcf29a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43561 } } }
I20260926 20:49:38.665480 9665 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:49:38.665494 9661 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6fe423fcbe142609ed354162fcf29a7 [sys.catalog]: This master's current role is: LEADER
I20260926 20:49:38.666028 9665 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:49:38.667014 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 9589
I20260926 20:49:38.667114 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/master-0/wal/instance
I20260926 20:49:38.667335 392 external_mini_cluster.cc:1428] Setting key cb5b96b5ea4a17e79f1fcc23521e6d8c
I20260926 20:49:38.667558 9665 catalog_manager.cc:1347] Generated new cluster ID: 61410ff5e93e401bbe763e8d6c40ea46
I20260926 20:49:38.667614 9665 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:49:38.672294 9679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:38.667925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40758 (local address 127.0.98.62:43561)
0926 20:49:38.668152 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:38.668159 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:38.668183 (+ 24us) server_negotiation.cc:407] Connection header received
0926 20:49:38.668270 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:38.668276 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:38.668333 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:38.668464 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:38.668689 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:38.669555 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:38.670328 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:38.670752 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:38.670878 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:38.671565 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:38.671568 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:38.671569 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:49:38.671783 (+ 214us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:38.671830 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:49:38.671841 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:38.671909 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:38.671993 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 20:49:38.672149 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":108,"threads_started":1}
I20260926 20:49:38.691596 9665 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:49:38.692340 9665 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:49:38.696918 9665 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6fe423fcbe142609ed354162fcf29a7: Generated new TSK 0
I20260926 20:49:38.697110 9665 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:49:38.698662 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43561
--builtin_ntp_servers=127.0.98.20:41907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 20:49:38.780860 9680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:38.781114 9680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:38.781147 9680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:38.782305 9680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:38.782368 9680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:38.782408 9680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:49:38.783779 9680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:38.784050 9680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:38.784343 9680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:38.790179 9688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:38.790151 9686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:38.790184 9685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:38.790436 9680 server_base.cc:1034] running on GCE node
I20260926 20:49:38.790670 9680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:38.790933 9680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:38.792088 9680 hybrid_clock.cc:648] HybridClock initialized: now 1790455778792073 us; error 32 us; skew 500 ppm
I20260926 20:49:38.793493 9680 webserver.cc:460] Webserver started at http://127.0.98.1:43657/ using document root <none> and password file <none>
I20260926 20:49:38.793751 9680 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:38.793797 9680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:38.793999 9680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:49:38.795079 9680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/ts-0/data/instance:
uuid: "e23be29f4221445ab1649920e3f93509"
format_stamp: "Formatted at 2026-09-26 20:49:38 on dist-test-slave-hwlj"
server_key: "d34256faf9d0c5847ec40ca7766fad68"
server_key_iv: "66efc1054a05edb853579f664fae6d6c"
server_key_version: "encryptionkey@0"
I20260926 20:49:38.795437 9680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e23be29f4221445ab1649920e3f93509"
format_stamp: "Formatted at 2026-09-26 20:49:38 on dist-test-slave-hwlj"
server_key: "d34256faf9d0c5847ec40ca7766fad68"
server_key_iv: "66efc1054a05edb853579f664fae6d6c"
server_key_version: "encryptionkey@0"
I20260926 20:49:38.796852 9680 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:49:38.797654 9694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:38.797803 9680 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 20:49:38.797892 9680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/ts-0/wal
uuid: "e23be29f4221445ab1649920e3f93509"
format_stamp: "Formatted at 2026-09-26 20:49:38 on dist-test-slave-hwlj"
server_key: "d34256faf9d0c5847ec40ca7766fad68"
server_key_iv: "66efc1054a05edb853579f664fae6d6c"
server_key_version: "encryptionkey@0"
I20260926 20:49:38.797973 9680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:38.813930 9680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:38.814245 9680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:38.814389 9680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:38.814606 9680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:49:38.814931 9680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:49:38.814992 9680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:38.815034 9680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:49:38.815062 9680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:38.821430 9680 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43631
I20260926 20:49:38.821481 9807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43631 every 8 connection(s)
I20260926 20:49:38.821841 9680 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:49:38.823879 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 9680
I20260926 20:49:38.823985 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/ts-0/wal/instance
I20260926 20:49:38.824209 392 external_mini_cluster.cc:1428] Setting key f9687cd0d3faefae54ee268d5c458742
I20260926 20:49:38.825126 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43561
--builtin_ntp_servers=127.0.98.20:41907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 20:49:38.827531 9679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:38.822625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52241 (local address 127.0.98.62:43561)
0926 20:49:38.822709 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:38.822712 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:38.822927 (+ 215us) server_negotiation.cc:407] Connection header received
0926 20:49:38.823437 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:38.823438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:38.823452 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:38.823502 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:38.824452 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:38.824982 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:38.826067 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:38.826283 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:38.826361 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:38.827233 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:38.827234 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:38.827234 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:49:38.827239 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:38.827254 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:49:38.827258 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:38.827304 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:38.827412 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 20:49:38.827448 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 20:49:38.827740 9810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:38.822598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43561 (local address 127.0.98.1:52241)
0926 20:49:38.822803 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:38.822838 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:38.823343 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:38.823564 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:38.823567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:38.823575 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:38.824330 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:38.824338 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:38.825081 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:38.825082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:38.825950 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:38.825952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:38.826368 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:38.826369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:38.826510 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:38.827074 (+ 564us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:49:38.827074 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:38.827104 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:49:38.827126 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:38.827304 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:38.827309 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:38.827312 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:49:38.827413 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 20:49:38.827514 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":103,"threads_started":1}
I20260926 20:49:38.828261 9808 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43561
I20260926 20:49:38.828372 9808 heartbeater.cc:464] Registering TS with master...
I20260926 20:49:38.828555 9808 heartbeater.cc:511] Master 127.0.98.62:43561 requested a full tablet report, sending...
I20260926 20:49:38.829140 9620 ts_manager.cc:194] Registered new tserver with Master: e23be29f4221445ab1649920e3f93509 (127.0.98.1:43631)
I20260926 20:49:38.830090 9620 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:52241
W20260926 20:49:38.907848 9811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:38.908066 9811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:38.908087 9811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:38.909194 9811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:38.909226 9811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:38.909250 9811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:49:38.910641 9811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:38.910838 9811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:38.911100 9811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:38.916213 9819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:38.916229 9817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:38.916222 9816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:38.916388 9811 server_base.cc:1034] running on GCE node
I20260926 20:49:38.916651 9811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:38.916865 9811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:38.918051 9811 hybrid_clock.cc:648] HybridClock initialized: now 1790455778918021 us; error 51 us; skew 500 ppm
I20260926 20:49:38.919339 9811 webserver.cc:460] Webserver started at http://127.0.98.2:37871/ using document root <none> and password file <none>
I20260926 20:49:38.919610 9811 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:38.919663 9811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:38.919826 9811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:49:38.920945 9811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/ts-1/data/instance:
uuid: "0caebd2c05044cf6aa260e21dcf55bf3"
format_stamp: "Formatted at 2026-09-26 20:49:38 on dist-test-slave-hwlj"
server_key: "bca73cb24d673998f67ebb6d4d792d1b"
server_key_iv: "6198e7c8be2e635393c99f9b15d09641"
server_key_version: "encryptionkey@0"
I20260926 20:49:38.921310 9811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0caebd2c05044cf6aa260e21dcf55bf3"
format_stamp: "Formatted at 2026-09-26 20:49:38 on dist-test-slave-hwlj"
server_key: "bca73cb24d673998f67ebb6d4d792d1b"
server_key_iv: "6198e7c8be2e635393c99f9b15d09641"
server_key_version: "encryptionkey@0"
I20260926 20:49:38.922631 9811 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:49:38.923393 9825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:38.923576 9811 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:49:38.923666 9811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/ts-1/wal
uuid: "0caebd2c05044cf6aa260e21dcf55bf3"
format_stamp: "Formatted at 2026-09-26 20:49:38 on dist-test-slave-hwlj"
server_key: "bca73cb24d673998f67ebb6d4d792d1b"
server_key_iv: "6198e7c8be2e635393c99f9b15d09641"
server_key_version: "encryptionkey@0"
I20260926 20:49:38.923770 9811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:38.939481 9811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:38.939898 9811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:38.940060 9811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:38.940317 9811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:49:38.940665 9811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:49:38.940721 9811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:38.940765 9811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:49:38.940794 9811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:38.946717 9811 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38151
I20260926 20:49:38.946766 9938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38151 every 8 connection(s)
I20260926 20:49:38.947149 9811 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
I20260926 20:49:38.950356 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 9811
I20260926 20:49:38.950472 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/ts-1/wal/instance
I20260926 20:49:38.950664 392 external_mini_cluster.cc:1428] Setting key 968d1698674d13b2dc54914767530731
I20260926 20:49:38.951689 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43561
--builtin_ntp_servers=127.0.98.20:41907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 20:49:38.952960 9679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:38.948010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53275 (local address 127.0.98.62:43561)
0926 20:49:38.948091 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:38.948094 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:38.948325 (+ 231us) server_negotiation.cc:407] Connection header received
0926 20:49:38.948833 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:38.948834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:38.948845 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:38.948914 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:38.949581 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:38.950084 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:38.951269 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:38.951552 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:38.951624 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:38.952611 (+ 987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:38.952613 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:38.952613 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:49:38.952618 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:38.952636 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:49:38.952641 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:38.952700 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:38.952765 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 20:49:38.952795 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 20:49:38.953095 9941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:38.947996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43561 (local address 127.0.98.2:53275)
0926 20:49:38.948202 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:38.948236 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:38.948733 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:38.948969 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:38.948973 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:38.948981 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:38.949495 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:38.949499 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:38.950168 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:38.950185 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:38.951201 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:38.951204 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:38.951641 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:38.951642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:38.951812 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:38.952462 (+ 650us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:49:38.952462 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:38.952491 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:49:38.952515 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:38.952708 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:38.952711 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:38.952714 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:49:38.952766 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 20:49:38.952849 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":105,"threads_started":1}
I20260926 20:49:38.953565 9939 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43561
I20260926 20:49:38.953673 9939 heartbeater.cc:464] Registering TS with master...
I20260926 20:49:38.953859 9939 heartbeater.cc:511] Master 127.0.98.62:43561 requested a full tablet report, sending...
I20260926 20:49:38.954317 9620 ts_manager.cc:194] Registered new tserver with Master: 0caebd2c05044cf6aa260e21dcf55bf3 (127.0.98.2:38151)
I20260926 20:49:38.954852 9620 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:53275
W20260926 20:49:39.032487 9942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:39.032677 9942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:39.032697 9942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:39.033742 9942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:39.033774 9942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:39.033798 9942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:49:39.035120 9942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:39.035310 9942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:39.035554 9942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:39.040654 9948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:39.040616 9950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:39.040869 9942 server_base.cc:1034] running on GCE node
W20260926 20:49:39.040616 9947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:39.041152 9942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:39.041360 9942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:39.042534 9942 hybrid_clock.cc:648] HybridClock initialized: now 1790455779042513 us; error 43 us; skew 500 ppm
I20260926 20:49:39.043951 9942 webserver.cc:460] Webserver started at http://127.0.98.3:35253/ using document root <none> and password file <none>
I20260926 20:49:39.044224 9942 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:39.044271 9942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:39.044441 9942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:49:39.045583 9942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/ts-2/data/instance:
uuid: "9da4e2ce35d64cba8d51132ecc0cfd25"
format_stamp: "Formatted at 2026-09-26 20:49:39 on dist-test-slave-hwlj"
server_key: "00f14dfaa675acb7935df2ac3129661e"
server_key_iv: "ec7a3ff064cec72ec8d9076ee2e58ef3"
server_key_version: "encryptionkey@0"
I20260926 20:49:39.045948 9942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9da4e2ce35d64cba8d51132ecc0cfd25"
format_stamp: "Formatted at 2026-09-26 20:49:39 on dist-test-slave-hwlj"
server_key: "00f14dfaa675acb7935df2ac3129661e"
server_key_iv: "ec7a3ff064cec72ec8d9076ee2e58ef3"
server_key_version: "encryptionkey@0"
I20260926 20:49:39.047298 9942 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:49:39.048174 9956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:39.048350 9942 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 20:49:39.048451 9942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/ts-2/wal
uuid: "9da4e2ce35d64cba8d51132ecc0cfd25"
format_stamp: "Formatted at 2026-09-26 20:49:39 on dist-test-slave-hwlj"
server_key: "00f14dfaa675acb7935df2ac3129661e"
server_key_iv: "ec7a3ff064cec72ec8d9076ee2e58ef3"
server_key_version: "encryptionkey@0"
I20260926 20:49:39.048534 9942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:39.059159 9942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:39.059438 9942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:39.059589 9942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:39.059883 9942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:49:39.060215 9942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:49:39.060276 9942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:39.060314 9942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:49:39.060343 9942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:39.066643 9942 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33857
I20260926 20:49:39.066694 10069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33857 every 8 connection(s)
I20260926 20:49:39.067058 9942 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
I20260926 20:49:39.067379 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 9942
I20260926 20:49:39.067485 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455725523895-392-0/minicluster-data/ts-2/wal/instance
I20260926 20:49:39.067677 392 external_mini_cluster.cc:1428] Setting key 2adb67d08c5f869db977d8861b034c34
I20260926 20:49:39.072312 9679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:39.067840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35189 (local address 127.0.98.62:43561)
0926 20:49:39.067988 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:39.067990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:39.068245 (+ 255us) server_negotiation.cc:407] Connection header received
0926 20:49:39.068971 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:39.068972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:39.068981 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:39.069039 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:39.069655 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:39.070144 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:39.071065 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:39.071250 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:39.071309 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:39.072098 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:39.072099 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:39.072100 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:49:39.072103 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:39.072117 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:49:39.072121 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:39.072168 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:39.072231 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 20:49:39.072255 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260926 20:49:39.072546 10072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:39.067855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43561 (local address 127.0.98.3:35189)
0926 20:49:39.068074 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:39.068117 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:39.068887 (+ 770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:39.069093 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:39.069096 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:39.069104 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:39.069575 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:39.069578 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:39.070221 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:39.070221 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:39.070990 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:39.070992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:39.071323 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:39.071324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:39.071426 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:39.071975 (+ 549us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:49:39.071975 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:39.072004 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:49:39.072024 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:39.072171 (+ 147us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:39.072173 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:39.072175 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 20:49:39.072221 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 20:49:39.072329 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":89,"threads_started":1}
I20260926 20:49:39.073043 10070 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43561
I20260926 20:49:39.073119 10070 heartbeater.cc:464] Registering TS with master...
I20260926 20:49:39.073309 10070 heartbeater.cc:511] Master 127.0.98.62:43561 requested a full tablet report, sending...
I20260926 20:49:39.073719 9620 ts_manager.cc:194] Registered new tserver with Master: 9da4e2ce35d64cba8d51132ecc0cfd25 (127.0.98.3:33857)
I20260926 20:49:39.074100 9620 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:35189
I20260926 20:49:39.079504 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 20:49:39.081629 9679 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:39.080912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40764 (local address 127.0.98.62:43561)
0926 20:49:39.080948 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:39.080950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:39.081172 (+ 222us) server_negotiation.cc:407] Connection header received
0926 20:49:39.081196 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:39.081196 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:39.081208 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:39.081263 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:39.081561 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40764: BlockingRecv error: recv got EOF from 127.0.0.1:40764 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 20:49:39.081669 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 9680
I20260926 20:49:39.086131 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 9811
I20260926 20:49:39.090410 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 9942
I20260926 20:49:39.094686 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 9589
2026-09-26T20:49:39Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (578 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T20:49:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:49:39Z Disabled control of system clock
I20260926 20:49:39.121073 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44175
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44175
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 20:49:39.204576 10081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:39.205035 10081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:39.205088 10081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:39.207120 10081 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:49:39.207170 10081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:39.207185 10081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:39.207197 10081 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:49:39.207211 10081 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:49:39.208806 10081 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44175
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:39.209039 10081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:39.209293 10081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:39.214157 10089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:39.214205 10081 server_base.cc:1034] running on GCE node
W20260926 20:49:39.214267 10087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:39.214184 10086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:39.214607 10081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:39.214910 10081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:39.216059 10081 hybrid_clock.cc:648] HybridClock initialized: now 1790455779216045 us; error 28 us; skew 500 ppm
I20260926 20:49:39.217247 10081 webserver.cc:460] Webserver started at http://127.0.98.62:43347/ using document root <none> and password file <none>
I20260926 20:49:39.217497 10081 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:39.217545 10081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:39.217711 10081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:49:39.218794 10081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/master-0/data/instance:
uuid: "57baa1eed8ec4c2e89c57c603b0b1681"
format_stamp: "Formatted at 2026-09-26 20:49:39 on dist-test-slave-hwlj"
server_key: "4490314f7b4beb4b5c627e681db50856"
server_key_iv: "bdc61153743b3b4ebd0de077f51e949f"
server_key_version: "encryptionkey@0"
I20260926 20:49:39.219151 10081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/master-0/wal/instance:
uuid: "57baa1eed8ec4c2e89c57c603b0b1681"
format_stamp: "Formatted at 2026-09-26 20:49:39 on dist-test-slave-hwlj"
server_key: "4490314f7b4beb4b5c627e681db50856"
server_key_iv: "bdc61153743b3b4ebd0de077f51e949f"
server_key_version: "encryptionkey@0"
I20260926 20:49:39.220569 10081 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:49:39.221370 10095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:39.221554 10081 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:49:39.221659 10081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "57baa1eed8ec4c2e89c57c603b0b1681"
format_stamp: "Formatted at 2026-09-26 20:49:39 on dist-test-slave-hwlj"
server_key: "4490314f7b4beb4b5c627e681db50856"
server_key_iv: "bdc61153743b3b4ebd0de077f51e949f"
server_key_version: "encryptionkey@0"
I20260926 20:49:39.221735 10081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:39.225117 10081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:39.225382 10081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:39.225521 10081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:39.229246 10081 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44175
I20260926 20:49:39.229278 10147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44175 every 8 connection(s)
I20260926 20:49:39.229614 10081 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:49:39.230294 10148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:49:39.233042 10148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57baa1eed8ec4c2e89c57c603b0b1681: Bootstrap starting.
I20260926 20:49:39.233565 10148 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57baa1eed8ec4c2e89c57c603b0b1681: Neither blocks nor log segments found. Creating new log.
I20260926 20:49:39.233804 10148 log.cc:824] T 00000000000000000000000000000000 P 57baa1eed8ec4c2e89c57c603b0b1681: Log is configured to *not* fsync() on all Append() calls
I20260926 20:49:39.234423 10148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57baa1eed8ec4c2e89c57c603b0b1681: No bootstrap required, opened a new log
I20260926 20:49:39.235283 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 10081
I20260926 20:49:39.235386 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/master-0/wal/instance
I20260926 20:49:39.235582 392 external_mini_cluster.cc:1428] Setting key 6eba1b655161c16176485442379f227c
I20260926 20:49:39.236415 10148 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57baa1eed8ec4c2e89c57c603b0b1681 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57baa1eed8ec4c2e89c57c603b0b1681" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44175 } }
I20260926 20:49:39.236579 10148 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57baa1eed8ec4c2e89c57c603b0b1681 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:49:39.236624 10148 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57baa1eed8ec4c2e89c57c603b0b1681 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57baa1eed8ec4c2e89c57c603b0b1681, State: Initialized, Role: FOLLOWER
I20260926 20:49:39.236742 10148 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57baa1eed8ec4c2e89c57c603b0b1681 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57baa1eed8ec4c2e89c57c603b0b1681" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44175 } }
I20260926 20:49:39.236843 10148 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57baa1eed8ec4c2e89c57c603b0b1681 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:49:39.236882 10148 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57baa1eed8ec4c2e89c57c603b0b1681 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:49:39.236923 10148 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57baa1eed8ec4c2e89c57c603b0b1681 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:49:39.237702 10148 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57baa1eed8ec4c2e89c57c603b0b1681 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57baa1eed8ec4c2e89c57c603b0b1681" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44175 } }
I20260926 20:49:39.237843 10148 leader_election.cc:302] T 00000000000000000000000000000000 P 57baa1eed8ec4c2e89c57c603b0b1681 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57baa1eed8ec4c2e89c57c603b0b1681; no voters:
I20260926 20:49:39.237847 10152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:39.236139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41166 (local address 127.0.98.62:44175)
0926 20:49:39.236341 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:39.236348 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:39.236372 (+ 24us) server_negotiation.cc:407] Connection header received
0926 20:49:39.236465 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:39.236470 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:39.236512 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:39.236656 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:39.237326 (+ 670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:39.237330 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:39.237330 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:49:39.237333 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:39.237368 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:49:39.237379 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:39.237433 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:39.237538 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 20:49:39.237616 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":93,"threads_started":1}
I20260926 20:49:39.238063 10148 leader_election.cc:288] T 00000000000000000000000000000000 P 57baa1eed8ec4c2e89c57c603b0b1681 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:49:39.238334 10148 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57baa1eed8ec4c2e89c57c603b0b1681 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:49:39.238333 10153 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57baa1eed8ec4c2e89c57c603b0b1681 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:49:39.238481 10153 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57baa1eed8ec4c2e89c57c603b0b1681 [term 1 LEADER]: Becoming Leader. State: Replica: 57baa1eed8ec4c2e89c57c603b0b1681, State: Running, Role: LEADER
I20260926 20:49:39.238584 10153 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57baa1eed8ec4c2e89c57c603b0b1681 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57baa1eed8ec4c2e89c57c603b0b1681" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44175 } }
I20260926 20:49:39.238911 10153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57baa1eed8ec4c2e89c57c603b0b1681 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57baa1eed8ec4c2e89c57c603b0b1681. Latest consensus state: current_term: 1 leader_uuid: "57baa1eed8ec4c2e89c57c603b0b1681" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57baa1eed8ec4c2e89c57c603b0b1681" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44175 } } }
I20260926 20:49:39.238921 10154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57baa1eed8ec4c2e89c57c603b0b1681 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57baa1eed8ec4c2e89c57c603b0b1681" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57baa1eed8ec4c2e89c57c603b0b1681" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44175 } } }
I20260926 20:49:39.239009 10153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57baa1eed8ec4c2e89c57c603b0b1681 [sys.catalog]: This master's current role is: LEADER
I20260926 20:49:39.239037 10154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57baa1eed8ec4c2e89c57c603b0b1681 [sys.catalog]: This master's current role is: LEADER
I20260926 20:49:39.239194 10162 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:49:39.239749 10162 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:49:39.241504 10162 catalog_manager.cc:1347] Generated new cluster ID: dcb2171a52334562a66f4d4db69e38e5
I20260926 20:49:39.241564 10162 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:49:39.248871 10162 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:49:39.249063 10162 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:49:39.256791 10162 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57baa1eed8ec4c2e89c57c603b0b1681: Generated new TSK 0
I20260926 20:49:39.257025 10162 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:49:39.258675 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44175
--builtin_ntp_servers=127.0.98.20:39437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 20:49:39.361413 10172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:39.361644 10172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:39.361667 10172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:39.362948 10172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:39.363005 10172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:39.363034 10172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:49:39.364588 10172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:39.364862 10172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:39.365188 10172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:39.371096 10178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:39.371168 10177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:39.371356 10180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:39.371619 10172 server_base.cc:1034] running on GCE node
I20260926 20:49:39.371889 10172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:39.372146 10172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:39.373318 10172 hybrid_clock.cc:648] HybridClock initialized: now 1790455779373295 us; error 40 us; skew 500 ppm
I20260926 20:49:39.378438 10172 webserver.cc:460] Webserver started at http://127.0.98.1:44859/ using document root <none> and password file <none>
I20260926 20:49:39.378710 10172 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:39.378780 10172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:39.378929 10172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:49:39.380147 10172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/ts-0/data/instance:
uuid: "1a671747342448f4bf720e5967fe73ec"
format_stamp: "Formatted at 2026-09-26 20:49:39 on dist-test-slave-hwlj"
server_key: "e92544bd8d94ed1f5a7d13eedf0572f7"
server_key_iv: "d4aeaf43607c30cba5e8788439c0c551"
server_key_version: "encryptionkey@0"
I20260926 20:49:39.380513 10172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1a671747342448f4bf720e5967fe73ec"
format_stamp: "Formatted at 2026-09-26 20:49:39 on dist-test-slave-hwlj"
server_key: "e92544bd8d94ed1f5a7d13eedf0572f7"
server_key_iv: "d4aeaf43607c30cba5e8788439c0c551"
server_key_version: "encryptionkey@0"
I20260926 20:49:39.381886 10172 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:49:39.382669 10186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:39.382848 10172 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:49:39.382949 10172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/ts-0/wal
uuid: "1a671747342448f4bf720e5967fe73ec"
format_stamp: "Formatted at 2026-09-26 20:49:39 on dist-test-slave-hwlj"
server_key: "e92544bd8d94ed1f5a7d13eedf0572f7"
server_key_iv: "d4aeaf43607c30cba5e8788439c0c551"
server_key_version: "encryptionkey@0"
I20260926 20:49:39.383054 10172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:39.386447 10172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:39.386714 10172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:39.386852 10172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:39.387064 10172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:49:39.387370 10172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:49:39.387423 10172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:39.387456 10172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:49:39.387486 10172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:39.394327 10172 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37667
I20260926 20:49:39.394371 10299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37667 every 8 connection(s)
I20260926 20:49:39.394709 10172 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:49:39.397881 10152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:39.395438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38317 (local address 127.0.98.62:44175)
0926 20:49:39.395510 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:39.395513 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:39.395809 (+ 296us) server_negotiation.cc:407] Connection header received
0926 20:49:39.396344 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:39.396345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:39.396351 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:39.396435 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:39.397617 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:39.397618 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:39.397618 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:49:39.397623 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:39.397640 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:49:39.397646 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:39.397713 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:39.397794 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 20:49:39.397810 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 20:49:39.398023 10302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:39.395427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44175 (local address 127.0.98.1:38317)
0926 20:49:39.395633 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:39.395674 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:39.396242 (+ 568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:39.396489 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:39.396493 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:39.396501 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:39.397456 (+ 955us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:49:39.397456 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:39.397485 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:49:39.397516 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:39.397716 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:39.397720 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:39.397723 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:49:39.397781 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 20:49:39.397830 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":109,"threads_started":1}
I20260926 20:49:39.398567 10300 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44175
I20260926 20:49:39.398685 10300 heartbeater.cc:464] Registering TS with master...
I20260926 20:49:39.398831 10300 heartbeater.cc:511] Master 127.0.98.62:44175 requested a full tablet report, sending...
I20260926 20:49:39.399317 10112 ts_manager.cc:194] Registered new tserver with Master: 1a671747342448f4bf720e5967fe73ec (127.0.98.1:37667)
I20260926 20:49:39.404219 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 10172
I20260926 20:49:39.404295 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/ts-0/wal/instance
I20260926 20:49:39.404441 392 external_mini_cluster.cc:1428] Setting key c30f6e97a7bec735705739c4f52f58dd
I20260926 20:49:39.405369 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44175
--builtin_ntp_servers=127.0.98.20:39437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 20:49:39.486999 10303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:39.487191 10303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:39.487214 10303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:39.488374 10303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:39.488413 10303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:39.488440 10303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:49:39.490356 10303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:39.490559 10303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:39.490795 10303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:39.495630 10309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:39.495576 10308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:39.495572 10311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:39.495929 10303 server_base.cc:1034] running on GCE node
I20260926 20:49:39.496285 10303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:39.496528 10303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:39.497699 10303 hybrid_clock.cc:648] HybridClock initialized: now 1790455779497680 us; error 28 us; skew 500 ppm
I20260926 20:49:39.498965 10303 webserver.cc:460] Webserver started at http://127.0.98.2:33761/ using document root <none> and password file <none>
I20260926 20:49:39.499221 10303 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:39.499290 10303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:39.499434 10303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:49:39.500610 10303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/ts-1/data/instance:
uuid: "cdf7def00e7c43daae3fb35a2e2ab455"
format_stamp: "Formatted at 2026-09-26 20:49:39 on dist-test-slave-hwlj"
server_key: "804f77ab5afd28ed7a203d6a19aae683"
server_key_iv: "653f920b6bd50cce5c26326d0260948c"
server_key_version: "encryptionkey@0"
I20260926 20:49:39.500989 10303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/ts-1/wal/instance:
uuid: "cdf7def00e7c43daae3fb35a2e2ab455"
format_stamp: "Formatted at 2026-09-26 20:49:39 on dist-test-slave-hwlj"
server_key: "804f77ab5afd28ed7a203d6a19aae683"
server_key_iv: "653f920b6bd50cce5c26326d0260948c"
server_key_version: "encryptionkey@0"
I20260926 20:49:39.502279 10303 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 20:49:39.503043 10317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:39.503229 10303 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 20:49:39.503326 10303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/ts-1/wal
uuid: "cdf7def00e7c43daae3fb35a2e2ab455"
format_stamp: "Formatted at 2026-09-26 20:49:39 on dist-test-slave-hwlj"
server_key: "804f77ab5afd28ed7a203d6a19aae683"
server_key_iv: "653f920b6bd50cce5c26326d0260948c"
server_key_version: "encryptionkey@0"
I20260926 20:49:39.503414 10303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:39.506892 10303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:39.507159 10303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:39.507301 10303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:39.507516 10303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:49:39.507867 10303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:49:39.507918 10303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:39.507957 10303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:49:39.507987 10303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:39.513731 10303 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35081
I20260926 20:49:39.513782 10430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35081 every 8 connection(s)
I20260926 20:49:39.514130 10303 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
I20260926 20:49:39.517184 10152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:39.514878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42867 (local address 127.0.98.62:44175)
0926 20:49:39.514924 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:39.514926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:39.515172 (+ 246us) server_negotiation.cc:407] Connection header received
0926 20:49:39.515688 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:39.515690 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:39.515697 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:39.515766 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:39.516911 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:39.516912 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:39.516913 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:49:39.516918 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:39.516934 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:49:39.516939 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:39.517004 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:39.517096 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 20:49:39.517112 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 20:49:39.517400 10433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:39.514857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44175 (local address 127.0.98.2:42867)
0926 20:49:39.515053 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:39.515085 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:39.515572 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:39.515848 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:39.515851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:39.515860 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:39.516763 (+ 903us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:49:39.516764 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:39.516791 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:49:39.516819 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:39.517004 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:39.517010 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:39.517012 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 20:49:39.517106 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 20:49:39.517167 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":85,"threads_started":1}
I20260926 20:49:39.517848 10431 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44175
I20260926 20:49:39.517922 10431 heartbeater.cc:464] Registering TS with master...
I20260926 20:49:39.518031 10431 heartbeater.cc:511] Master 127.0.98.62:44175 requested a full tablet report, sending...
I20260926 20:49:39.518435 10112 ts_manager.cc:194] Registered new tserver with Master: cdf7def00e7c43daae3fb35a2e2ab455 (127.0.98.2:35081)
I20260926 20:49:39.519951 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 10303
I20260926 20:49:39.520081 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/ts-1/wal/instance
I20260926 20:49:39.520287 392 external_mini_cluster.cc:1428] Setting key aa655d8170d702c7500a17403380cca9
I20260926 20:49:39.521216 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44175
--builtin_ntp_servers=127.0.98.20:39437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 20:49:39.601548 10434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:39.601750 10434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:39.601776 10434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:39.602870 10434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:39.602903 10434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:39.602928 10434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:49:39.604199 10434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:39.604382 10434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:39.604614 10434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:39.610131 10440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:39.610123 10439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:39.610134 10442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:39.610314 10434 server_base.cc:1034] running on GCE node
I20260926 20:49:39.610549 10434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:39.610808 10434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:39.611969 10434 hybrid_clock.cc:648] HybridClock initialized: now 1790455779611959 us; error 30 us; skew 500 ppm
I20260926 20:49:39.613353 10434 webserver.cc:460] Webserver started at http://127.0.98.3:44265/ using document root <none> and password file <none>
I20260926 20:49:39.613612 10434 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:39.613684 10434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:39.613842 10434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:49:39.614974 10434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/ts-2/data/instance:
uuid: "e2982393d83a49798953b8b1f6d2b213"
format_stamp: "Formatted at 2026-09-26 20:49:39 on dist-test-slave-hwlj"
server_key: "c2c9c57000d5bc20d46ecadc223c01e8"
server_key_iv: "86eb90a1066a8ee26eb27f8379abfcce"
server_key_version: "encryptionkey@0"
I20260926 20:49:39.615345 10434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e2982393d83a49798953b8b1f6d2b213"
format_stamp: "Formatted at 2026-09-26 20:49:39 on dist-test-slave-hwlj"
server_key: "c2c9c57000d5bc20d46ecadc223c01e8"
server_key_iv: "86eb90a1066a8ee26eb27f8379abfcce"
server_key_version: "encryptionkey@0"
I20260926 20:49:39.616787 10434 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:49:39.617588 10448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:39.617782 10434 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:49:39.617889 10434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/ts-2/wal
uuid: "e2982393d83a49798953b8b1f6d2b213"
format_stamp: "Formatted at 2026-09-26 20:49:39 on dist-test-slave-hwlj"
server_key: "c2c9c57000d5bc20d46ecadc223c01e8"
server_key_iv: "86eb90a1066a8ee26eb27f8379abfcce"
server_key_version: "encryptionkey@0"
I20260926 20:49:39.617973 10434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:39.621335 10434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:39.621615 10434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:39.621759 10434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:39.621963 10434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:49:39.622263 10434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:49:39.622316 10434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:39.622357 10434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:49:39.622385 10434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:39.628196 10434 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46493
I20260926 20:49:39.628270 10561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46493 every 8 connection(s)
I20260926 20:49:39.628608 10434 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
I20260926 20:49:39.631736 10152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:39.629369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47779 (local address 127.0.98.62:44175)
0926 20:49:39.629415 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:39.629417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:39.629659 (+ 242us) server_negotiation.cc:407] Connection header received
0926 20:49:39.630191 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:39.630193 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:39.630199 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:39.630257 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:39.631448 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:39.631450 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:39.631450 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:49:39.631455 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:39.631469 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:49:39.631474 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:39.631551 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:39.631648 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 20:49:39.631665 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 20:49:39.631960 10564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:39.629347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44175 (local address 127.0.98.3:47779)
0926 20:49:39.629534 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:39.629565 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:39.630075 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:39.630358 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:39.630361 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:39.630369 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:39.631279 (+ 910us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:49:39.631279 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:39.631307 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:49:39.631334 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:39.631548 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:39.631554 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:39.631558 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:49:39.631656 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 20:49:39.631718 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":76,"threads_started":1}
I20260926 20:49:39.632432 10562 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44175
I20260926 20:49:39.632521 10562 heartbeater.cc:464] Registering TS with master...
I20260926 20:49:39.632627 10562 heartbeater.cc:511] Master 127.0.98.62:44175 requested a full tablet report, sending...
I20260926 20:49:39.632946 10112 ts_manager.cc:194] Registered new tserver with Master: e2982393d83a49798953b8b1f6d2b213 (127.0.98.3:46493)
I20260926 20:49:39.635955 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 10434
I20260926 20:49:39.636030 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455725523895-392-0/minicluster-data/ts-2/wal/instance
I20260926 20:49:39.636171 392 external_mini_cluster.cc:1428] Setting key e8e3ef5a2aff960afe44e0f608162bc2
I20260926 20:49:39.636915 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 20:49:39.638810 10152 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:49:39.638297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47356 (local address 127.0.98.62:44175)
0926 20:49:39.638334 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:39.638337 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:39.638445 (+ 108us) server_negotiation.cc:407] Connection header received
0926 20:49:39.638524 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:39.638525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:39.638530 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:39.638586 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:39.638720 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47356: BlockingRecv error: recv got EOF from 127.0.0.1:47356 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
I20260926 20:49:39.638876 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 10172
I20260926 20:49:39.643513 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 10303
I20260926 20:49:39.648278 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 10434
I20260926 20:49:39.653296 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 10081
2026-09-26T20:49:39Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (559 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:49:39 dist-test-slave-hwlj krb5kdc[10572](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:49:39 dist-test-slave-hwlj krb5kdc[10572](info): set up 2 sockets
Sep 26 20:49:39 dist-test-slave-hwlj krb5kdc[10572](info): commencing operation
krb5kdc: starting...
W20260926 20:49:40.692878 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/krb5kdc/test-user.keytab.
Sep 26 20:49:40 dist-test-slave-hwlj krb5kdc[10572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:49:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:49:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 20:49:40.818593 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32927
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32927
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:49:40.898959 10585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:40.899159 10585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:40.899181 10585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:40.900360 10585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:49:40.900403 10585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:40.900417 10585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:40.900429 10585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:49:40.900472 10585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:49:40.901976 10585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32927
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:40.902235 10585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:40.902467 10585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:40.907483 10590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:40.907450 10591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:40.907455 10593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:40.907609 10585 server_base.cc:1034] running on GCE node
I20260926 20:49:40.908017 10585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:40.908327 10585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:40.909492 10585 hybrid_clock.cc:648] HybridClock initialized: now 1790455780909481 us; error 30 us; skew 500 ppm
Sep 26 20:49:40 dist-test-slave-hwlj krb5kdc[10572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455780, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:49:40.912087 10585 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:49:40.912572 10585 webserver.cc:460] Webserver started at http://127.0.98.62:38163/ using document root <none> and password file <none>
I20260926 20:49:40.912829 10585 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:40.912874 10585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:40.913010 10585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:49:40.914076 10585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/master-0/data/instance:
uuid: "4770918e48334d6d91d192d09410e4af"
format_stamp: "Formatted at 2026-09-26 20:49:40 on dist-test-slave-hwlj"
server_key: "2a280b7aca0abb0ab2afa19a1b385db5"
server_key_iv: "ce1aa66bf33d3b87ff30dede6cd4f686"
server_key_version: "encryptionkey@0"
I20260926 20:49:40.914438 10585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/master-0/wal/instance:
uuid: "4770918e48334d6d91d192d09410e4af"
format_stamp: "Formatted at 2026-09-26 20:49:40 on dist-test-slave-hwlj"
server_key: "2a280b7aca0abb0ab2afa19a1b385db5"
server_key_iv: "ce1aa66bf33d3b87ff30dede6cd4f686"
server_key_version: "encryptionkey@0"
I20260926 20:49:40.915882 10585 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:49:40.916759 10600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:40.917086 10585 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:49:40.917187 10585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "4770918e48334d6d91d192d09410e4af"
format_stamp: "Formatted at 2026-09-26 20:49:40 on dist-test-slave-hwlj"
server_key: "2a280b7aca0abb0ab2afa19a1b385db5"
server_key_iv: "ce1aa66bf33d3b87ff30dede6cd4f686"
server_key_version: "encryptionkey@0"
I20260926 20:49:40.917264 10585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:40.934680 10585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:40.937148 10585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:40.937301 10585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:40.941460 10585 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32927
I20260926 20:49:40.941509 10652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32927 every 8 connection(s)
I20260926 20:49:40.941900 10585 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:49:40.942866 10653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:49:40.943368 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 10585
I20260926 20:49:40.943485 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/master-0/wal/instance
I20260926 20:49:40.943668 392 external_mini_cluster.cc:1428] Setting key 00022150e020912098858bb03112779f
I20260926 20:49:40.946100 10653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4770918e48334d6d91d192d09410e4af: Bootstrap starting.
I20260926 20:49:40.946699 10653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4770918e48334d6d91d192d09410e4af: Neither blocks nor log segments found. Creating new log.
I20260926 20:49:40.947000 10653 log.cc:824] T 00000000000000000000000000000000 P 4770918e48334d6d91d192d09410e4af: Log is configured to *not* fsync() on all Append() calls
I20260926 20:49:40.947683 10653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4770918e48334d6d91d192d09410e4af: No bootstrap required, opened a new log
I20260926 20:49:40.949460 10653 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4770918e48334d6d91d192d09410e4af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4770918e48334d6d91d192d09410e4af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32927 } }
I20260926 20:49:40.949625 10653 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4770918e48334d6d91d192d09410e4af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:49:40.949672 10653 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4770918e48334d6d91d192d09410e4af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4770918e48334d6d91d192d09410e4af, State: Initialized, Role: FOLLOWER
I20260926 20:49:40.949834 10653 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4770918e48334d6d91d192d09410e4af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4770918e48334d6d91d192d09410e4af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32927 } }
I20260926 20:49:40.949942 10653 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4770918e48334d6d91d192d09410e4af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:49:40.949993 10653 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4770918e48334d6d91d192d09410e4af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:49:40.950042 10653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4770918e48334d6d91d192d09410e4af [term 0 FOLLOWER]: Advancing to term 1
Sep 26 20:49:40 dist-test-slave-hwlj krb5kdc[10572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:49:40.950646 10653 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4770918e48334d6d91d192d09410e4af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4770918e48334d6d91d192d09410e4af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32927 } }
I20260926 20:49:40.950765 10653 leader_election.cc:302] T 00000000000000000000000000000000 P 4770918e48334d6d91d192d09410e4af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4770918e48334d6d91d192d09410e4af; no voters:
I20260926 20:49:40.950932 10653 leader_election.cc:288] T 00000000000000000000000000000000 P 4770918e48334d6d91d192d09410e4af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:49:40.950966 10658 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4770918e48334d6d91d192d09410e4af [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:49:40.951150 10658 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4770918e48334d6d91d192d09410e4af [term 1 LEADER]: Becoming Leader. State: Replica: 4770918e48334d6d91d192d09410e4af, State: Running, Role: LEADER
I20260926 20:49:40.951231 10653 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4770918e48334d6d91d192d09410e4af [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:49:40.951231 10658 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4770918e48334d6d91d192d09410e4af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4770918e48334d6d91d192d09410e4af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32927 } }
I20260926 20:49:40.951915 10659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4770918e48334d6d91d192d09410e4af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4770918e48334d6d91d192d09410e4af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4770918e48334d6d91d192d09410e4af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32927 } } }
I20260926 20:49:40.951941 10660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4770918e48334d6d91d192d09410e4af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4770918e48334d6d91d192d09410e4af. Latest consensus state: current_term: 1 leader_uuid: "4770918e48334d6d91d192d09410e4af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4770918e48334d6d91d192d09410e4af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32927 } } }
I20260926 20:49:40.952013 10659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4770918e48334d6d91d192d09410e4af [sys.catalog]: This master's current role is: LEADER
I20260926 20:49:40.952023 10660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4770918e48334d6d91d192d09410e4af [sys.catalog]: This master's current role is: LEADER
I20260926 20:49:40.952493 10665 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:49:40.953075 10665 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:49:40.954618 10665 catalog_manager.cc:1347] Generated new cluster ID: 92f0d3aff1644a6bb91a0a06fbcf5a8e
I20260926 20:49:40.954679 10665 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:49:40.954882 10656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:40.944233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53896 (local address 127.0.98.62:32927)
0926 20:49:40.944426 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:40.944434 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:40.944460 (+ 26us) server_negotiation.cc:407] Connection header received
0926 20:49:40.944889 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:40.944896 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:40.944944 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:40.945062 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:40.945858 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:40.946810 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:40.947633 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:40.948132 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:40.948261 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:40.950683 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:40.950687 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:40.950688 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:40.950699 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:40.952936 (+ 2237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:40.953377 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:40.953381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:40.953381 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:40.953498 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:40.953710 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:40.953711 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:40.953711 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:40.953963 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:40.954133 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:40.954383 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 20:49:40.954521 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":100,"threads_started":1}
I20260926 20:49:40.967031 10665 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:49:40.967789 10665 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:49:40.977167 10665 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4770918e48334d6d91d192d09410e4af: Generated new TSK 0
I20260926 20:49:40.977375 10665 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 20:49:41.001971 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32927
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:49:41.083031 10679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:41.083235 10679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:41.083258 10679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:41.084388 10679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:41.084424 10679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:41.084456 10679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:49:41.085840 10679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:41.086074 10679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:41.086254 10679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:49:41.091183 10679 server_base.cc:1034] running on GCE node
W20260926 20:49:41.091157 10685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:41.091166 10687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:41.091166 10684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:41.091619 10679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:41.091881 10679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:41.093046 10679 hybrid_clock.cc:648] HybridClock initialized: now 1790455781093025 us; error 76 us; skew 500 ppm
Sep 26 20:49:41 dist-test-slave-hwlj krb5kdc[10572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455781, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:49:41.095549 10679 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 20:49:41.096019 10679 webserver.cc:460] Webserver started at http://127.0.98.1:42229/ using document root <none> and password file <none>
I20260926 20:49:41.096246 10679 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:41.096313 10679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:41.096469 10679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:49:41.097498 10679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/ts-0/data/instance:
uuid: "36ba21c14ef54ea1a0c442c9c7e1b3de"
format_stamp: "Formatted at 2026-09-26 20:49:41 on dist-test-slave-hwlj"
server_key: "5fee51db6dd52dfdcf18bb5f51fbe31f"
server_key_iv: "e5ee9da8354750820dd71c9be3ac1a54"
server_key_version: "encryptionkey@0"
I20260926 20:49:41.097857 10679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/ts-0/wal/instance:
uuid: "36ba21c14ef54ea1a0c442c9c7e1b3de"
format_stamp: "Formatted at 2026-09-26 20:49:41 on dist-test-slave-hwlj"
server_key: "5fee51db6dd52dfdcf18bb5f51fbe31f"
server_key_iv: "e5ee9da8354750820dd71c9be3ac1a54"
server_key_version: "encryptionkey@0"
I20260926 20:49:41.099169 10679 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 20:49:41.100030 10694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:41.100208 10679 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:49:41.100307 10679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/ts-0/wal
uuid: "36ba21c14ef54ea1a0c442c9c7e1b3de"
format_stamp: "Formatted at 2026-09-26 20:49:41 on dist-test-slave-hwlj"
server_key: "5fee51db6dd52dfdcf18bb5f51fbe31f"
server_key_iv: "e5ee9da8354750820dd71c9be3ac1a54"
server_key_version: "encryptionkey@0"
I20260926 20:49:41.100391 10679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:41.114945 10679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:41.117074 10679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:41.117246 10679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:41.117496 10679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:49:41.117825 10679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:49:41.117887 10679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:41.117934 10679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:49:41.117956 10679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:41.123855 10679 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43189
I20260926 20:49:41.123939 10807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43189 every 8 connection(s)
I20260926 20:49:41.124274 10679 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:49:41.126668 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 10679
I20260926 20:49:41.126787 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/ts-0/wal/instance
I20260926 20:49:41.127012 392 external_mini_cluster.cc:1428] Setting key 75c47bf147ff07d7e53291757bd1c935
Sep 26 20:49:41 dist-test-slave-hwlj krb5kdc[10572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455781, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:49:41.134625 10656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:41.124994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40005 (local address 127.0.98.62:32927)
0926 20:49:41.125070 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:41.125072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:41.125308 (+ 236us) server_negotiation.cc:407] Connection header received
0926 20:49:41.125813 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:41.125815 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:41.125825 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:41.125884 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:41.126920 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:41.127678 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:41.128759 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:41.128992 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:41.129066 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:41.131439 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:41.131441 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:41.131441 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:41.131457 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:41.133191 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:41.133609 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:41.133610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:41.133611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:41.133682 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:41.133934 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:41.133935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:41.133935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:41.134087 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:41.134163 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:41.134410 (+ 247us) server_negotiation.cc:299] Negotiation successful
0926 20:49:41.134528 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 20:49:41.134658 10810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:41.124997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32927 (local address 127.0.98.1:40005)
0926 20:49:41.125200 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:41.125236 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:41.125694 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:41.125972 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:41.125976 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:41.126552 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:41.126744 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:41.126749 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:41.127805 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:41.127806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:41.128647 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:41.128649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:41.129073 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:41.129074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:41.129208 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:41.129771 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:41.129779 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:41.131330 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:41.133278 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:41.133284 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:41.133286 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:41.133530 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:41.133758 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:41.133759 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:41.133759 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:41.133859 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:41.134157 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:41.134162 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:41.134277 (+ 115us) client_negotiation.cc:769] Sending connection context
0926 20:49:41.134358 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 20:49:41.134494 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":97,"threads_started":1}
I20260926 20:49:41.135367 10808 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32927
I20260926 20:49:41.135481 10808 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 20:49:41.135692 10808 heartbeater.cc:511] Master 127.0.98.62:32927 requested a full tablet report, sending...
I20260926 20:49:41.136257 10617 ts_manager.cc:194] Registered new tserver with Master: 36ba21c14ef54ea1a0c442c9c7e1b3de (127.0.98.1:43189)
I20260926 20:49:41.137040 10617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40005
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 20:49:41.155709 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32927
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:49:41.238566 10813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:41.238793 10813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:41.238818 10813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:41.240118 10813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:41.240165 10813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:41.240193 10813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:49:41.241693 10813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:41.241959 10813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:41.242213 10813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:41.247344 10819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:41.247331 10818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:41.247367 10813 server_base.cc:1034] running on GCE node
W20260926 20:49:41.247323 10821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:41.247857 10813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:41.248135 10813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:41.249302 10813 hybrid_clock.cc:648] HybridClock initialized: now 1790455781249294 us; error 42 us; skew 500 ppm
Sep 26 20:49:41 dist-test-slave-hwlj krb5kdc[10572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455781, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:49:41.252149 10813 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 20:49:41.252628 10813 webserver.cc:460] Webserver started at http://127.0.98.2:41587/ using document root <none> and password file <none>
I20260926 20:49:41.252866 10813 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:41.252936 10813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:41.253108 10813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:49:41.254190 10813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/ts-1/data/instance:
uuid: "be814ae520ab4ef0a9a2d2608707e900"
format_stamp: "Formatted at 2026-09-26 20:49:41 on dist-test-slave-hwlj"
server_key: "86c0987ff2dcb691b3220ad4e899b1a8"
server_key_iv: "c1ca60e752eff8cb45e32026a2e321a7"
server_key_version: "encryptionkey@0"
I20260926 20:49:41.254565 10813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/ts-1/wal/instance:
uuid: "be814ae520ab4ef0a9a2d2608707e900"
format_stamp: "Formatted at 2026-09-26 20:49:41 on dist-test-slave-hwlj"
server_key: "86c0987ff2dcb691b3220ad4e899b1a8"
server_key_iv: "c1ca60e752eff8cb45e32026a2e321a7"
server_key_version: "encryptionkey@0"
I20260926 20:49:41.256073 10813 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:49:41.256903 10828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:41.257081 10813 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:49:41.257181 10813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/ts-1/wal
uuid: "be814ae520ab4ef0a9a2d2608707e900"
format_stamp: "Formatted at 2026-09-26 20:49:41 on dist-test-slave-hwlj"
server_key: "86c0987ff2dcb691b3220ad4e899b1a8"
server_key_iv: "c1ca60e752eff8cb45e32026a2e321a7"
server_key_version: "encryptionkey@0"
I20260926 20:49:41.257266 10813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:41.278568 10813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:41.281628 10813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:41.281791 10813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:41.282055 10813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:49:41.282384 10813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:49:41.282445 10813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:41.282488 10813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:49:41.282516 10813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:41.288579 10813 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35449
I20260926 20:49:41.288621 10941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35449 every 8 connection(s)
I20260926 20:49:41.288951 10813 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
I20260926 20:49:41.290611 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 10813
I20260926 20:49:41.290750 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/ts-1/wal/instance
I20260926 20:49:41.290959 392 external_mini_cluster.cc:1428] Setting key aceab255d8f69cbb990820fec2b39b82
Sep 26 20:49:41 dist-test-slave-hwlj krb5kdc[10572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455781, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:49:41.299001 10656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:41.289650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44747 (local address 127.0.98.62:32927)
0926 20:49:41.289698 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:41.289700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:41.289881 (+ 181us) server_negotiation.cc:407] Connection header received
0926 20:49:41.290408 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:41.290410 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:41.290422 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:41.290472 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:41.291566 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:41.292170 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:41.293164 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:41.293387 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:41.293468 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:41.295817 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:41.295819 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:41.295819 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:41.295828 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:41.297287 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:41.297892 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:41.297893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:41.297893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:41.297952 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:41.298282 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:41.298283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:41.298283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:41.298403 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:41.298468 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:41.298861 (+ 393us) server_negotiation.cc:299] Negotiation successful
0926 20:49:41.298935 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 20:49:41.299237 10944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:41.289602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32927 (local address 127.0.98.2:44747)
0926 20:49:41.289748 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:41.289796 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:41.290303 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:41.290543 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:41.290547 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:41.291264 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:41.291467 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:41.291473 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:41.292265 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:41.292266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:41.293071 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:41.293073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:41.293476 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:41.293477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:41.293606 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:41.294188 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:41.294197 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:41.295686 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:41.297387 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:41.297395 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:41.297398 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:41.297723 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:41.298035 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:41.298036 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:41.298036 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:41.298185 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:41.298491 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:41.298498 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:41.298682 (+ 184us) client_negotiation.cc:769] Sending connection context
0926 20:49:41.298808 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 20:49:41.298972 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":73,"threads_started":1}
I20260926 20:49:41.299710 10942 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32927
I20260926 20:49:41.299919 10942 heartbeater.cc:464] Registering TS with master...
I20260926 20:49:41.300125 10942 heartbeater.cc:511] Master 127.0.98.62:32927 requested a full tablet report, sending...
I20260926 20:49:41.300544 10617 ts_manager.cc:194] Registered new tserver with Master: be814ae520ab4ef0a9a2d2608707e900 (127.0.98.2:35449)
I20260926 20:49:41.300930 10617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44747
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 20:49:41.319535 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32927
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:49:41.399166 10947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:49:41.399415 10947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:49:41.399452 10947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:49:41.400607 10947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:49:41.400676 10947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:49:41.400718 10947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:49:41.402137 10947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:49:41.402395 10947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:49:41.402627 10947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:49:41.407836 10953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:41.407828 10955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:49:41.407832 10952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:49:41.408059 10947 server_base.cc:1034] running on GCE node
I20260926 20:49:41.408281 10947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:49:41.408521 10947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:49:41.409686 10947 hybrid_clock.cc:648] HybridClock initialized: now 1790455781409672 us; error 26 us; skew 500 ppm
Sep 26 20:49:41 dist-test-slave-hwlj krb5kdc[10572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455781, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:49:41.412323 10947 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 20:49:41.412752 10947 webserver.cc:460] Webserver started at http://127.0.98.3:41253/ using document root <none> and password file <none>
I20260926 20:49:41.412988 10947 fs_manager.cc:359] Metadata directory not provided
I20260926 20:49:41.413057 10947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:49:41.413213 10947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:49:41.414336 10947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/ts-2/data/instance:
uuid: "f5eaa83c29e44546843645bb029971ba"
format_stamp: "Formatted at 2026-09-26 20:49:41 on dist-test-slave-hwlj"
server_key: "a0551bbee21de4cd9fddb5451428d5b2"
server_key_iv: "b5e64483d9cc027965657385881815ea"
server_key_version: "encryptionkey@0"
I20260926 20:49:41.414698 10947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f5eaa83c29e44546843645bb029971ba"
format_stamp: "Formatted at 2026-09-26 20:49:41 on dist-test-slave-hwlj"
server_key: "a0551bbee21de4cd9fddb5451428d5b2"
server_key_iv: "b5e64483d9cc027965657385881815ea"
server_key_version: "encryptionkey@0"
I20260926 20:49:41.416090 10947 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:49:41.416836 10962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:41.417037 10947 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:49:41.417145 10947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/ts-2/wal
uuid: "f5eaa83c29e44546843645bb029971ba"
format_stamp: "Formatted at 2026-09-26 20:49:41 on dist-test-slave-hwlj"
server_key: "a0551bbee21de4cd9fddb5451428d5b2"
server_key_iv: "b5e64483d9cc027965657385881815ea"
server_key_version: "encryptionkey@0"
I20260926 20:49:41.417220 10947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:49:41.427618 10947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:49:41.429535 10947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:49:41.429703 10947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:49:41.429929 10947 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:49:41.430246 10947 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:49:41.430301 10947 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:41.430341 10947 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:49:41.430369 10947 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:49:41.436230 10947 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36893
I20260926 20:49:41.436285 11075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36893 every 8 connection(s)
I20260926 20:49:41.436615 10947 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 20:49:41 dist-test-slave-hwlj krb5kdc[10572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455781, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:49:41.444128 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 10947
I20260926 20:49:41.444243 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/ts-2/wal/instance
I20260926 20:49:41.444461 392 external_mini_cluster.cc:1428] Setting key 8a7f3194c837cee7b5f79f6f3e02ff98
I20260926 20:49:41.446666 10656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:41.437368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53125 (local address 127.0.98.62:32927)
0926 20:49:41.437460 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:41.437463 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:41.437672 (+ 209us) server_negotiation.cc:407] Connection header received
0926 20:49:41.438195 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:41.438198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:41.438210 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:41.438259 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:41.439209 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:41.440137 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:41.441142 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:41.441356 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:41.441442 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:41.443601 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:41.443604 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:41.443604 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:41.443612 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:41.445514 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:41.445884 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:41.445884 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:41.445885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:41.445933 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:41.446150 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:41.446151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:41.446151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:41.446249 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:41.446297 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:41.446546 (+ 249us) server_negotiation.cc:299] Negotiation successful
0926 20:49:41.446603 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 20:49:41.446903 11078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:41.437274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32927 (local address 127.0.98.3:53125)
0926 20:49:41.437523 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:41.437573 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:41.438087 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:41.438337 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:41.438340 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:41.438907 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:49:41.439106 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:41.439111 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:41.440229 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:41.440230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:41.441043 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:41.441045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:41.441456 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:41.441458 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:41.441585 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:41.442059 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:49:41.442067 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:49:41.443498 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:49:41.445589 (+ 2091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:41.445594 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:41.445596 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:41.445819 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:41.445997 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:49:41.445998 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:49:41.445998 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:49:41.446094 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:49:41.446303 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:49:41.446321 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:49:41.446437 (+ 116us) client_negotiation.cc:769] Sending connection context
0926 20:49:41.446519 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 20:49:41.446639 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":105,"threads_started":1}
I20260926 20:49:41.447341 11076 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32927
I20260926 20:49:41.447461 11076 heartbeater.cc:464] Registering TS with master...
I20260926 20:49:41.447661 11076 heartbeater.cc:511] Master 127.0.98.62:32927 requested a full tablet report, sending...
I20260926 20:49:41.448042 10617 ts_manager.cc:194] Registered new tserver with Master: f5eaa83c29e44546843645bb029971ba (127.0.98.3:36893)
I20260926 20:49:41.448431 10617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53125
I20260926 20:49:41.456228 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:49:41.463244 10656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:41.457438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53898 (local address 127.0.98.62:32927)
0926 20:49:41.457494 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:41.457496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:41.457601 (+ 105us) server_negotiation.cc:407] Connection header received
0926 20:49:41.457650 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:41.457651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:41.457660 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:41.457699 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:49:41.458241 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:41.458718 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:41.459507 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:41.459683 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:41.459734 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:41.461062 (+ 1328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:49:41.461064 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:49:41.461064 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:49:41.461069 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:49:41.462284 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:41.462619 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:41.462620 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:41.462621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:41.462671 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:49:41.462811 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:49:41.462812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:49:41.462812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:49:41.462942 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:49:41.463018 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:41.463126 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 20:49:41.463184 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 20:49:41.464778 10617 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53898:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:49:41.466353 10617 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:49:41.472914 11087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:41.468615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33302 (local address 127.0.98.3:36893)
0926 20:49:41.468773 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:41.468777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:41.468784 (+ 7us) server_negotiation.cc:407] Connection header received
0926 20:49:41.468814 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:41.468815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:41.468828 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:41.468912 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:49:41.469070 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:41.470209 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:41.471865 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:41.472558 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:41.472619 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:41.472665 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:41.472789 (+ 124us) server_negotiation.cc:299] Negotiation successful
0926 20:49:41.472830 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":87,"threads_started":1}
I20260926 20:49:41.472929 11084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:41.468358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36893 (local address 127.0.0.1:33302)
0926 20:49:41.468463 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:41.468504 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:41.468573 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:41.468926 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:41.468930 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:41.468933 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:49:41.468996 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:41.468997 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:41.470332 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:41.470334 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:41.471793 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:41.471796 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:41.472628 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:41.472628 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:41.472677 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:41.472687 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 20:49:41.472763 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 20:49:41.472830 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"mutex_wait_us":117,"thread_start_us":57,"threads_started":1}
I20260926 20:49:41.473161 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:41.469032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38842 (local address 127.0.98.1:43189)
0926 20:49:41.469303 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:41.469307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:41.469316 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:49:41.469351 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:41.469354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:41.469369 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:41.469442 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:49:41.469649 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:41.470652 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:41.471764 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:41.472496 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:41.472577 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:41.472642 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:41.473045 (+ 403us) server_negotiation.cc:299] Negotiation successful
0926 20:49:41.473087 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":81,"threads_started":1}
I20260926 20:49:41.472937 11085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:41.468374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43189 (local address 127.0.0.1:38842)
0926 20:49:41.468586 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:41.468595 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:41.468636 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:41.469520 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:41.469521 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:41.469522 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:49:41.469563 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:41.469564 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:41.470753 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:41.470754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:41.471662 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:41.471664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:41.472575 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:41.472576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:41.472669 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:41.472685 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 20:49:41.472763 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 20:49:41.472830 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":25,"thread_start_us":55,"threads_started":1}
I20260926 20:49:41.473807 11086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:41.468361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35449 (local address 127.0.0.1:39958)
0926 20:49:41.468519 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:41.468530 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:41.468581 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:41.469326 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:41.469327 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:41.469329 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:49:41.469377 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:41.469378 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:41.470330 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:41.470331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:41.471793 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:41.471796 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:41.473584 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:41.473585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:41.473655 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:41.473659 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:49:41.473694 (+ 35us) client_negotiation.cc:240] Negotiation successful
0926 20:49:41.473731 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":85,"thread_start_us":69,"threads_started":1}
I20260926 20:49:41.473747 11010 tablet_service.cc:1467] Processing CreateTablet for tablet 1395f57adbf24443aa5ecb418ed6ece0 (DEFAULT_TABLE table=test-table [id=a79d42fcfa7340bdb4078d466ff3469b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:49:41.474144 11010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1395f57adbf24443aa5ecb418ed6ece0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:49:41.474169 11088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:41.468797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39958 (local address 127.0.98.2:35449)
0926 20:49:41.469151 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:41.469156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:41.469165 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:49:41.469203 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:41.469205 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:41.469221 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:41.469462 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:49:41.469527 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:41.470221 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:41.472709 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:41.473491 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:41.473915 (+ 424us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:41.474002 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:41.474023 (+ 21us) server_negotiation.cc:299] Negotiation successful
0926 20:49:41.474056 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":92,"threads_started":1}
I20260926 20:49:41.474210 10742 tablet_service.cc:1467] Processing CreateTablet for tablet 1395f57adbf24443aa5ecb418ed6ece0 (DEFAULT_TABLE table=test-table [id=a79d42fcfa7340bdb4078d466ff3469b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:49:41.474500 10742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1395f57adbf24443aa5ecb418ed6ece0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:49:41.474803 10876 tablet_service.cc:1467] Processing CreateTablet for tablet 1395f57adbf24443aa5ecb418ed6ece0 (DEFAULT_TABLE table=test-table [id=a79d42fcfa7340bdb4078d466ff3469b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:49:41.475073 10876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1395f57adbf24443aa5ecb418ed6ece0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:49:41.477680 11092 tablet_bootstrap.cc:492] T 1395f57adbf24443aa5ecb418ed6ece0 P be814ae520ab4ef0a9a2d2608707e900: Bootstrap starting.
I20260926 20:49:41.477910 11091 tablet_bootstrap.cc:492] T 1395f57adbf24443aa5ecb418ed6ece0 P 36ba21c14ef54ea1a0c442c9c7e1b3de: Bootstrap starting.
I20260926 20:49:41.478386 11092 tablet_bootstrap.cc:654] T 1395f57adbf24443aa5ecb418ed6ece0 P be814ae520ab4ef0a9a2d2608707e900: Neither blocks nor log segments found. Creating new log.
I20260926 20:49:41.478533 11091 tablet_bootstrap.cc:654] T 1395f57adbf24443aa5ecb418ed6ece0 P 36ba21c14ef54ea1a0c442c9c7e1b3de: Neither blocks nor log segments found. Creating new log.
I20260926 20:49:41.479007 11092 log.cc:824] T 1395f57adbf24443aa5ecb418ed6ece0 P be814ae520ab4ef0a9a2d2608707e900: Log is configured to *not* fsync() on all Append() calls
I20260926 20:49:41.479081 11091 log.cc:824] T 1395f57adbf24443aa5ecb418ed6ece0 P 36ba21c14ef54ea1a0c442c9c7e1b3de: Log is configured to *not* fsync() on all Append() calls
I20260926 20:49:41.479377 11090 tablet_bootstrap.cc:492] T 1395f57adbf24443aa5ecb418ed6ece0 P f5eaa83c29e44546843645bb029971ba: Bootstrap starting.
I20260926 20:49:41.479812 11092 tablet_bootstrap.cc:492] T 1395f57adbf24443aa5ecb418ed6ece0 P be814ae520ab4ef0a9a2d2608707e900: No bootstrap required, opened a new log
I20260926 20:49:41.479884 11092 ts_tablet_manager.cc:1397] T 1395f57adbf24443aa5ecb418ed6ece0 P be814ae520ab4ef0a9a2d2608707e900: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:49:41.480127 11091 tablet_bootstrap.cc:492] T 1395f57adbf24443aa5ecb418ed6ece0 P 36ba21c14ef54ea1a0c442c9c7e1b3de: No bootstrap required, opened a new log
I20260926 20:49:41.480194 11091 ts_tablet_manager.cc:1397] T 1395f57adbf24443aa5ecb418ed6ece0 P 36ba21c14ef54ea1a0c442c9c7e1b3de: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:49:41.480222 11090 tablet_bootstrap.cc:654] T 1395f57adbf24443aa5ecb418ed6ece0 P f5eaa83c29e44546843645bb029971ba: Neither blocks nor log segments found. Creating new log.
I20260926 20:49:41.480682 11090 log.cc:824] T 1395f57adbf24443aa5ecb418ed6ece0 P f5eaa83c29e44546843645bb029971ba: Log is configured to *not* fsync() on all Append() calls
I20260926 20:49:41.481353 11090 tablet_bootstrap.cc:492] T 1395f57adbf24443aa5ecb418ed6ece0 P f5eaa83c29e44546843645bb029971ba: No bootstrap required, opened a new log
I20260926 20:49:41.481434 11090 ts_tablet_manager.cc:1397] T 1395f57adbf24443aa5ecb418ed6ece0 P f5eaa83c29e44546843645bb029971ba: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:49:41.481892 11092 raft_consensus.cc:348] T 1395f57adbf24443aa5ecb418ed6ece0 P be814ae520ab4ef0a9a2d2608707e900 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eaa83c29e44546843645bb029971ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36893 } } peers { permanent_uuid: "be814ae520ab4ef0a9a2d2608707e900" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35449 } } peers { permanent_uuid: "36ba21c14ef54ea1a0c442c9c7e1b3de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43189 } }
I20260926 20:49:41.482095 11092 raft_consensus.cc:374] T 1395f57adbf24443aa5ecb418ed6ece0 P be814ae520ab4ef0a9a2d2608707e900 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:49:41.482133 11092 raft_consensus.cc:729] T 1395f57adbf24443aa5ecb418ed6ece0 P be814ae520ab4ef0a9a2d2608707e900 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be814ae520ab4ef0a9a2d2608707e900, State: Initialized, Role: FOLLOWER
I20260926 20:49:41.482061 11091 raft_consensus.cc:348] T 1395f57adbf24443aa5ecb418ed6ece0 P 36ba21c14ef54ea1a0c442c9c7e1b3de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eaa83c29e44546843645bb029971ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36893 } } peers { permanent_uuid: "be814ae520ab4ef0a9a2d2608707e900" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35449 } } peers { permanent_uuid: "36ba21c14ef54ea1a0c442c9c7e1b3de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43189 } }
I20260926 20:49:41.482205 11091 raft_consensus.cc:374] T 1395f57adbf24443aa5ecb418ed6ece0 P 36ba21c14ef54ea1a0c442c9c7e1b3de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:49:41.482244 11091 raft_consensus.cc:729] T 1395f57adbf24443aa5ecb418ed6ece0 P 36ba21c14ef54ea1a0c442c9c7e1b3de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36ba21c14ef54ea1a0c442c9c7e1b3de, State: Initialized, Role: FOLLOWER
I20260926 20:49:41.482259 11092 consensus_queue.cc:260] T 1395f57adbf24443aa5ecb418ed6ece0 P be814ae520ab4ef0a9a2d2608707e900 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eaa83c29e44546843645bb029971ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36893 } } peers { permanent_uuid: "be814ae520ab4ef0a9a2d2608707e900" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35449 } } peers { permanent_uuid: "36ba21c14ef54ea1a0c442c9c7e1b3de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43189 } }
I20260926 20:49:41.482370 11091 consensus_queue.cc:260] T 1395f57adbf24443aa5ecb418ed6ece0 P 36ba21c14ef54ea1a0c442c9c7e1b3de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eaa83c29e44546843645bb029971ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36893 } } peers { permanent_uuid: "be814ae520ab4ef0a9a2d2608707e900" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35449 } } peers { permanent_uuid: "36ba21c14ef54ea1a0c442c9c7e1b3de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43189 } }
I20260926 20:49:41.482610 10942 heartbeater.cc:503] Master 127.0.98.62:32927 was elected leader, sending a full tablet report...
I20260926 20:49:41.482643 11091 ts_tablet_manager.cc:1428] T 1395f57adbf24443aa5ecb418ed6ece0 P 36ba21c14ef54ea1a0c442c9c7e1b3de: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 20:49:41.482758 10808 heartbeater.cc:503] Master 127.0.98.62:32927 was elected leader, sending a full tablet report...
I20260926 20:49:41.482545 11092 ts_tablet_manager.cc:1428] T 1395f57adbf24443aa5ecb418ed6ece0 P be814ae520ab4ef0a9a2d2608707e900: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:49:41.483273 11090 raft_consensus.cc:348] T 1395f57adbf24443aa5ecb418ed6ece0 P f5eaa83c29e44546843645bb029971ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eaa83c29e44546843645bb029971ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36893 } } peers { permanent_uuid: "be814ae520ab4ef0a9a2d2608707e900" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35449 } } peers { permanent_uuid: "36ba21c14ef54ea1a0c442c9c7e1b3de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43189 } }
I20260926 20:49:41.483454 11090 raft_consensus.cc:374] T 1395f57adbf24443aa5ecb418ed6ece0 P f5eaa83c29e44546843645bb029971ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:49:41.483498 11090 raft_consensus.cc:729] T 1395f57adbf24443aa5ecb418ed6ece0 P f5eaa83c29e44546843645bb029971ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5eaa83c29e44546843645bb029971ba, State: Initialized, Role: FOLLOWER
I20260926 20:49:41.483615 11090 consensus_queue.cc:260] T 1395f57adbf24443aa5ecb418ed6ece0 P f5eaa83c29e44546843645bb029971ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eaa83c29e44546843645bb029971ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36893 } } peers { permanent_uuid: "be814ae520ab4ef0a9a2d2608707e900" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35449 } } peers { permanent_uuid: "36ba21c14ef54ea1a0c442c9c7e1b3de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43189 } }
I20260926 20:49:41.483927 11090 ts_tablet_manager.cc:1428] T 1395f57adbf24443aa5ecb418ed6ece0 P f5eaa83c29e44546843645bb029971ba: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260926 20:49:41.483930 11076 heartbeater.cc:503] Master 127.0.98.62:32927 was elected leader, sending a full tablet report...
W20260926 20:49:41.539506 10943 tablet.cc:2369] T 1395f57adbf24443aa5ecb418ed6ece0 P be814ae520ab4ef0a9a2d2608707e900: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:49:41.566921 11097 raft_consensus.cc:482] T 1395f57adbf24443aa5ecb418ed6ece0 P 36ba21c14ef54ea1a0c442c9c7e1b3de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:49:41.567051 11097 raft_consensus.cc:504] T 1395f57adbf24443aa5ecb418ed6ece0 P 36ba21c14ef54ea1a0c442c9c7e1b3de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eaa83c29e44546843645bb029971ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36893 } } peers { permanent_uuid: "be814ae520ab4ef0a9a2d2608707e900" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35449 } } peers { permanent_uuid: "36ba21c14ef54ea1a0c442c9c7e1b3de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43189 } }
I20260926 20:49:41.567437 11097 leader_election.cc:288] T 1395f57adbf24443aa5ecb418ed6ece0 P 36ba21c14ef54ea1a0c442c9c7e1b3de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5eaa83c29e44546843645bb029971ba (127.0.98.3:36893), be814ae520ab4ef0a9a2d2608707e900 (127.0.98.2:35449)
I20260926 20:49:41.570361 11087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:41.567713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35321 (local address 127.0.98.3:36893)
0926 20:49:41.567783 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:41.567785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:41.567794 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:49:41.567828 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:41.567829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:41.567842 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:41.567915 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:49:41.568093 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:41.568584 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:41.569609 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:41.569983 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:41.570077 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:41.570087 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:41.570256 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 20:49:41.570296 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 20:49:41.570362 10810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:41.567603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36893 (local address 127.0.98.1:35321)
0926 20:49:41.567666 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:41.567678 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:41.567729 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:41.567933 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:41.567935 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:41.567936 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:49:41.568021 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:41.568022 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:41.568675 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:41.568676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:41.569508 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:41.569511 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:41.570074 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:41.570075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:41.570172 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:41.570175 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:49:41.570244 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 20:49:41.570286 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":37}
I20260926 20:49:41.570456 11088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:41.567695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60971 (local address 127.0.98.2:35449)
0926 20:49:41.567794 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:41.567796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:41.567968 (+ 172us) server_negotiation.cc:407] Connection header received
0926 20:49:41.568055 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:41.568056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:41.568068 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:41.568138 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:49:41.568287 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:41.568747 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:41.569692 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:41.570071 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:41.570155 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:41.570166 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:41.570323 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 20:49:41.570359 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 20:49:41.570695 11099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:41.567641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35449 (local address 127.0.98.1:60971)
0926 20:49:41.567871 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:41.567880 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:49:41.567984 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:49:41.568143 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:49:41.568143 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:49:41.568145 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:49:41.568217 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:41.568218 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:41.568834 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:41.568835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:41.569597 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:49:41.569599 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:49:41.570156 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:49:41.570158 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:49:41.570239 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:41.570242 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:49:41.570305 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 20:49:41.570343 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":2,"thread_start_us":139,"threads_started":1}
I20260926 20:49:41.570746 11030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1395f57adbf24443aa5ecb418ed6ece0" candidate_uuid: "36ba21c14ef54ea1a0c442c9c7e1b3de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5eaa83c29e44546843645bb029971ba" is_pre_election: true
I20260926 20:49:41.570920 11030 raft_consensus.cc:2454] T 1395f57adbf24443aa5ecb418ed6ece0 P f5eaa83c29e44546843645bb029971ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36ba21c14ef54ea1a0c442c9c7e1b3de in term 0.
I20260926 20:49:41.571051 10896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1395f57adbf24443aa5ecb418ed6ece0" candidate_uuid: "36ba21c14ef54ea1a0c442c9c7e1b3de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be814ae520ab4ef0a9a2d2608707e900" is_pre_election: true
I20260926 20:49:41.571169 10695 leader_election.cc:302] T 1395f57adbf24443aa5ecb418ed6ece0 P 36ba21c14ef54ea1a0c442c9c7e1b3de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36ba21c14ef54ea1a0c442c9c7e1b3de, f5eaa83c29e44546843645bb029971ba; no voters:
I20260926 20:49:41.571229 10896 raft_consensus.cc:2454] T 1395f57adbf24443aa5ecb418ed6ece0 P be814ae520ab4ef0a9a2d2608707e900 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36ba21c14ef54ea1a0c442c9c7e1b3de in term 0.
I20260926 20:49:41.571317 11097 raft_consensus.cc:2781] T 1395f57adbf24443aa5ecb418ed6ece0 P 36ba21c14ef54ea1a0c442c9c7e1b3de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:49:41.571391 11097 raft_consensus.cc:482] T 1395f57adbf24443aa5ecb418ed6ece0 P 36ba21c14ef54ea1a0c442c9c7e1b3de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:49:41.571417 11097 raft_consensus.cc:3037] T 1395f57adbf24443aa5ecb418ed6ece0 P 36ba21c14ef54ea1a0c442c9c7e1b3de [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:49:41.572221 11097 raft_consensus.cc:504] T 1395f57adbf24443aa5ecb418ed6ece0 P 36ba21c14ef54ea1a0c442c9c7e1b3de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eaa83c29e44546843645bb029971ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36893 } } peers { permanent_uuid: "be814ae520ab4ef0a9a2d2608707e900" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35449 } } peers { permanent_uuid: "36ba21c14ef54ea1a0c442c9c7e1b3de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43189 } }
I20260926 20:49:41.572371 11097 leader_election.cc:288] T 1395f57adbf24443aa5ecb418ed6ece0 P 36ba21c14ef54ea1a0c442c9c7e1b3de [CANDIDATE]: Term 1 election: Requested vote from peers f5eaa83c29e44546843645bb029971ba (127.0.98.3:36893), be814ae520ab4ef0a9a2d2608707e900 (127.0.98.2:35449)
I20260926 20:49:41.572525 10896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1395f57adbf24443aa5ecb418ed6ece0" candidate_uuid: "36ba21c14ef54ea1a0c442c9c7e1b3de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be814ae520ab4ef0a9a2d2608707e900"
I20260926 20:49:41.572530 11030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1395f57adbf24443aa5ecb418ed6ece0" candidate_uuid: "36ba21c14ef54ea1a0c442c9c7e1b3de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5eaa83c29e44546843645bb029971ba"
I20260926 20:49:41.572623 11030 raft_consensus.cc:3037] T 1395f57adbf24443aa5ecb418ed6ece0 P f5eaa83c29e44546843645bb029971ba [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:49:41.572598 10896 raft_consensus.cc:3037] T 1395f57adbf24443aa5ecb418ed6ece0 P be814ae520ab4ef0a9a2d2608707e900 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:49:41.573369 11030 raft_consensus.cc:2454] T 1395f57adbf24443aa5ecb418ed6ece0 P f5eaa83c29e44546843645bb029971ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36ba21c14ef54ea1a0c442c9c7e1b3de in term 1.
I20260926 20:49:41.573387 10896 raft_consensus.cc:2454] T 1395f57adbf24443aa5ecb418ed6ece0 P be814ae520ab4ef0a9a2d2608707e900 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36ba21c14ef54ea1a0c442c9c7e1b3de in term 1.
I20260926 20:49:41.573546 10695 leader_election.cc:302] T 1395f57adbf24443aa5ecb418ed6ece0 P 36ba21c14ef54ea1a0c442c9c7e1b3de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36ba21c14ef54ea1a0c442c9c7e1b3de, f5eaa83c29e44546843645bb029971ba; no voters:
I20260926 20:49:41.573658 11097 raft_consensus.cc:2781] T 1395f57adbf24443aa5ecb418ed6ece0 P 36ba21c14ef54ea1a0c442c9c7e1b3de [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:49:41.573865 11097 raft_consensus.cc:686] T 1395f57adbf24443aa5ecb418ed6ece0 P 36ba21c14ef54ea1a0c442c9c7e1b3de [term 1 LEADER]: Becoming Leader. State: Replica: 36ba21c14ef54ea1a0c442c9c7e1b3de, State: Running, Role: LEADER
I20260926 20:49:41.573971 11097 consensus_queue.cc:237] T 1395f57adbf24443aa5ecb418ed6ece0 P 36ba21c14ef54ea1a0c442c9c7e1b3de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eaa83c29e44546843645bb029971ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36893 } } peers { permanent_uuid: "be814ae520ab4ef0a9a2d2608707e900" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35449 } } peers { permanent_uuid: "36ba21c14ef54ea1a0c442c9c7e1b3de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43189 } }
I20260926 20:49:41.574889 10616 catalog_manager.cc:5495] T 1395f57adbf24443aa5ecb418ed6ece0 P 36ba21c14ef54ea1a0c442c9c7e1b3de reported cstate change: term changed from 0 to 1, leader changed from <none> to 36ba21c14ef54ea1a0c442c9c7e1b3de (127.0.98.1). New cstate: current_term: 1 leader_uuid: "36ba21c14ef54ea1a0c442c9c7e1b3de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eaa83c29e44546843645bb029971ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be814ae520ab4ef0a9a2d2608707e900" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36ba21c14ef54ea1a0c442c9c7e1b3de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43189 } health_report { overall_health: HEALTHY } } }
I20260926 20:49:41.587803 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:41.585645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38846 (local address 127.0.98.1:43189)
0926 20:49:41.585704 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 20:49:41.585706 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:49:41.585715 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:49:41.585757 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:49:41.585758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:49:41.585775 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:49:41.585835 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:49:41.585951 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:41.586391 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:41.587100 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:49:41.587275 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:49:41.587323 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:49:41.587451 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:49:41.587544 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:49:41.587605 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 20:49:41.587637 (+ 32us) server_negotiation.cc:299] Negotiation successful
0926 20:49:41.587654 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 20:49:41.589422 11030 raft_consensus.cc:1261] T 1395f57adbf24443aa5ecb418ed6ece0 P f5eaa83c29e44546843645bb029971ba [term 1 FOLLOWER]: Refusing update from remote peer 36ba21c14ef54ea1a0c442c9c7e1b3de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:49:41.589465 10896 raft_consensus.cc:1261] T 1395f57adbf24443aa5ecb418ed6ece0 P be814ae520ab4ef0a9a2d2608707e900 [term 1 FOLLOWER]: Refusing update from remote peer 36ba21c14ef54ea1a0c442c9c7e1b3de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:49:41.589766 11097 consensus_queue.cc:1035] T 1395f57adbf24443aa5ecb418ed6ece0 P 36ba21c14ef54ea1a0c442c9c7e1b3de [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5eaa83c29e44546843645bb029971ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:49:41.589924 11100 consensus_queue.cc:1035] T 1395f57adbf24443aa5ecb418ed6ece0 P 36ba21c14ef54ea1a0c442c9c7e1b3de [LEADER]: Connected to new peer: Peer: permanent_uuid: "be814ae520ab4ef0a9a2d2608707e900" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:49:41.592226 11103 mvcc.cc:205] Tried to move back new op lower bound from 7333706881387671552 to 7333706881327570944. Current Snapshot: MvccSnapshot[applied={T|T < 7333706881387671552}]
I20260926 20:49:41.592983 11104 mvcc.cc:205] Tried to move back new op lower bound from 7333706881387671552 to 7333706881327570944. Current Snapshot: MvccSnapshot[applied={T|T < 7333706881387671552}]
I20260926 20:49:41.593211 11105 mvcc.cc:205] Tried to move back new op lower bound from 7333706881387671552 to 7333706881327570944. Current Snapshot: MvccSnapshot[applied={T|T < 7333706881387671552 or (T in {7333706881387671552})}]
I20260926 20:49:41.594498 10617 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:49:41.594584 10617 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:49:41.595299 10617 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4770918e48334d6d91d192d09410e4af: Sending DeleteTablet for 3 replicas of tablet 1395f57adbf24443aa5ecb418ed6ece0
I20260926 20:49:41.595598 11010 tablet_service.cc:1514] Processing DeleteTablet for tablet 1395f57adbf24443aa5ecb418ed6ece0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:49:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33302
I20260926 20:49:41.595602 10742 tablet_service.cc:1514] Processing DeleteTablet for tablet 1395f57adbf24443aa5ecb418ed6ece0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:49:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38842
I20260926 20:49:41.595628 10876 tablet_service.cc:1514] Processing DeleteTablet for tablet 1395f57adbf24443aa5ecb418ed6ece0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:49:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39958
I20260926 20:49:41.596076 11113 tablet_replica.cc:330] T 1395f57adbf24443aa5ecb418ed6ece0 P f5eaa83c29e44546843645bb029971ba: stopping tablet replica
I20260926 20:49:41.596114 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 10679
I20260926 20:49:41.595906 11114 tablet_replica.cc:330] T 1395f57adbf24443aa5ecb418ed6ece0 P be814ae520ab4ef0a9a2d2608707e900: stopping tablet replica
I20260926 20:49:41.596982 11114 raft_consensus.cc:2229] T 1395f57adbf24443aa5ecb418ed6ece0 P be814ae520ab4ef0a9a2d2608707e900 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:49:41.597081 11114 raft_consensus.cc:2258] T 1395f57adbf24443aa5ecb418ed6ece0 P be814ae520ab4ef0a9a2d2608707e900 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:49:41.597186 11113 raft_consensus.cc:2229] T 1395f57adbf24443aa5ecb418ed6ece0 P f5eaa83c29e44546843645bb029971ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:49:41.597285 11113 raft_consensus.cc:2258] T 1395f57adbf24443aa5ecb418ed6ece0 P f5eaa83c29e44546843645bb029971ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:49:41.597625 11113 ts_tablet_manager.cc:1905] T 1395f57adbf24443aa5ecb418ed6ece0 P f5eaa83c29e44546843645bb029971ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:49:41.597625 11114 ts_tablet_manager.cc:1905] T 1395f57adbf24443aa5ecb418ed6ece0 P be814ae520ab4ef0a9a2d2608707e900: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:49:41.599304 11114 ts_tablet_manager.cc:1918] T 1395f57adbf24443aa5ecb418ed6ece0 P be814ae520ab4ef0a9a2d2608707e900: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:49:41.599363 11114 log.cc:1196] T 1395f57adbf24443aa5ecb418ed6ece0 P be814ae520ab4ef0a9a2d2608707e900: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/ts-1/wal/wals/1395f57adbf24443aa5ecb418ed6ece0
I20260926 20:49:41.599637 11114 ts_tablet_manager.cc:1939] T 1395f57adbf24443aa5ecb418ed6ece0 P be814ae520ab4ef0a9a2d2608707e900: Deleting consensus metadata
I20260926 20:49:41.600277 10604 catalog_manager.cc:4841] TS be814ae520ab4ef0a9a2d2608707e900 (127.0.98.2:35449): tablet 1395f57adbf24443aa5ecb418ed6ece0 (table test-table [id=a79d42fcfa7340bdb4078d466ff3469b]) successfully deleted
I20260926 20:49:41.601433 11113 ts_tablet_manager.cc:1918] T 1395f57adbf24443aa5ecb418ed6ece0 P f5eaa83c29e44546843645bb029971ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:49:41.601507 11113 log.cc:1196] T 1395f57adbf24443aa5ecb418ed6ece0 P f5eaa83c29e44546843645bb029971ba: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455725523895-392-0/minicluster-data/ts-2/wal/wals/1395f57adbf24443aa5ecb418ed6ece0
I20260926 20:49:41.601821 11113 ts_tablet_manager.cc:1939] T 1395f57adbf24443aa5ecb418ed6ece0 P f5eaa83c29e44546843645bb029971ba: Deleting consensus metadata
I20260926 20:49:41.602234 10601 catalog_manager.cc:4841] TS f5eaa83c29e44546843645bb029971ba (127.0.98.3:36893): tablet 1395f57adbf24443aa5ecb418ed6ece0 (table test-table [id=a79d42fcfa7340bdb4078d466ff3469b]) successfully deleted
W20260926 20:49:41.602268 10602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43189 (error 108)
I20260926 20:49:41.602517 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 10813
I20260926 20:49:41.602720 11086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:49:41.602598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43189 (local address 127.0.0.1:38850)
0926 20:49:41.602648 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 20:49:41.602681 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 20:49:41.603199 10602 catalog_manager.cc:4584] TS 36ba21c14ef54ea1a0c442c9c7e1b3de (127.0.98.1:43189): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1395f57adbf24443aa5ecb418ed6ece0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43189: connect: Connection refused (error 111)
I20260926 20:49:41.607647 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 10947
I20260926 20:49:41.612018 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 10585
2026-09-26T20:49:41Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1959 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455725523895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:49:41 dist-test-slave-hwlj krb5kdc[11119](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:49:41 dist-test-slave-hwlj krb5kdc[11119](info): set up 2 sockets
Sep 26 20:49:41 dist-test-slave-hwlj krb5kdc[11119](info): commencing operation
krb5kdc: starting...
W20260926 20:49:42.648658 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455725523895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455725523895-392-0/krb5kdc/test-user.keytab.
Sep 26 20:49:42 dist-test-slave-hwlj krb5kdc[11119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:49:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:49:42Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455725523895-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455725523895-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455725523895-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455725523895-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455725523895-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455725523895-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260926 20:49:42.819214 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455725523895-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task7paSjL/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455725523895-392-0/minicluster-data/postgres -l logfile start
2026-09-26 20:49:45.444 UTC [11149] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 20:49:45.444 UTC [11149] LOG: listening on IPv4 address "127.0.98.20", port 37507
2026-09-26 20:49:45.463 UTC [11149] LOG: listening on Unix socket "/tmp/.s.PGSQL.37507"
2026-09-26 20:49:45.487 UTC [11152] LOG: database system was shut down at 2026-09-26 20:49:43 UTC
2026-09-26 20:49:45.490 UTC [11149] LOG: database system is ready to accept connections
I20260926 20:49:46.436774 392 mini_postgres.cc:96] Postgres bound to 37507
2026-09-26 20:49:46.442 UTC [11160] FATAL: database "slave" does not exist
127.0.98.20:37507 - accepting connections
I20260926 20:49:46.443241 392 mini_ranger.cc:161] Starting Ranger...
I20260926 20:49:46.457980 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 20:49:46.972726 392 mini_ranger.cc:87] Created ranger Postgres database
I20260926 20:49:46.972815 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455725523895-392-0/minicluster-data/ranger-admin
2026-09-26 20:49:47,242 [I] DB FLAVOR :POSTGRES
2026-09-26 20:49:47,243 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 20:49:47,243 [I] Checking connection..
2026-09-26 20:49:47,243 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7paSjL/build/release/bin/postgresql.jar:/tmp/dist-test-task7paSjL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37507/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 20:49:47,514 [I] Checking connection passed.
2026-09-26 20:49:47,515 [I] --------- Verifying version history table ---------
2026-09-26 20:49:47,515 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7paSjL/build/release/bin/postgresql.jar:/tmp/dist-test-task7paSjL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37507/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 20:49:47,771 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 20:49:47,772 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7paSjL/build/release/bin/postgresql.jar:/tmp/dist-test-task7paSjL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37507/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 20:49:48,032 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 20:49:48,032 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 20:49:48,032 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7paSjL/build/release/bin/postgresql.jar:/tmp/dist-test-task7paSjL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37507/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7paSjL/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 20:49:48.264 UTC [11224] WARNING: there is no transaction in progress
2026-09-26 20:49:48,271 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 20:49:48,271 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7paSjL/build/release/bin/postgresql.jar:/tmp/dist-test-task7paSjL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37507/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 20:49:48,502 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 20:49:48,502 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 20:49:48,502 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7paSjL/build/release/bin/postgresql.jar:/tmp/dist-test-task7paSjL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37507/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 20:49:48,745 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7paSjL/build/release/bin/postgresql.jar:/tmp/dist-test-task7paSjL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37507/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 20:49:48,977 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7paSjL/build/release/bin/postgresql.jar:/tmp/dist-test-task7paSjL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37507/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-hwlj.c.gcp-upstream.internal','N') ;"
2026-09-26 20:49:49,196 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7paSjL/build/release/bin/postgresql.jar:/tmp/dist-test-task7paSjL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37507/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 20:49:49,446 [I] Table x_portal_user does not exist in database ranger
2026-09-26 20:49:49,447 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7paSjL/build/release/bin/postgresql.jar:/tmp/dist-test-task7paSjL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37507/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 20:49:49,712 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 20:49:49,712 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7paSjL/build/release/bin/postgresql.jar:/tmp/dist-test-task7paSjL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37507/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 20:49:49,959 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7paSjL/build/release/bin/postgresql.jar:/tmp/dist-test-task7paSjL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37507/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 20:49:50,215 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 20:49:50,215 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7paSjL/build/release/bin/postgresql.jar:/tmp/dist-test-task7paSjL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37507/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7paSjL/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 20:49:51.188 UTC [11332] WARNING: there is no transaction in progress
2026-09-26 20:49:51.204 UTC [11332] WARNING: there is no transaction in progress
2026-09-26 20:49:51.221 UTC [11332] WARNING: there is no transaction in progress
2026-09-26 20:49:51.237 UTC [11332] WARNING: there is no transaction in progress
2026-09-26 20:49:51.253 UTC [11332] WARNING: there is no transaction in progress
2026-09-26 20:49:51.275 UTC [11332] WARNING: there is no transaction in progress
2026-09-26 20:49:51.334 UTC [11332] WARNING: there is no transaction in progress
2026-09-26 20:49:51.341 UTC [11332] WARNING: there is no transaction in progress
2026-09-26 20:49:51.356 UTC [11332] WARNING: there is no transaction in progress
2026-09-26 20:49:51.366 UTC [11332] WARNING: there is no transaction in progress
2026-09-26 20:49:51.376 UTC [11332] WARNING: there is no transaction in progress
2026-09-26 20:49:51.389 UTC [11332] WARNING: there is no transaction in progress
2026-09-26 20:49:51.399 UTC [11332] WARNING: there is no transaction in progress
2026-09-26 20:49:51.405 UTC [11332] WARNING: there is no transaction in progress
2026-09-26 20:49:51.412 UTC [11332] WARNING: there is no transaction in progress
2026-09-26 20:49:51.418 UTC [11332] WARNING: there is no transaction in progress
2026-09-26 20:49:52,084 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 20:49:52,085 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7paSjL/build/release/bin/postgresql.jar:/tmp/dist-test-task7paSjL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37507/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 20:49:52,325 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 20:49:52,325 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7paSjL/build/release/bin/postgresql.jar:/tmp/dist-test-task7paSjL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37507/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 20:49:52,564 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 20:49:52,564 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7paSjL/build/release/bin/postgresql.jar:/tmp/dist-test-task7paSjL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37507/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 20:49:52,834 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 20:49:52,835 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7paSjL/build/release/bin/postgresql.jar:/tmp/dist-test-task7paSjL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37507/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 20:49:53,063 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7paSjL/build/release/bin/postgresql.jar:/tmp/dist-test-task7paSjL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37507/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 20:49:53,309 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7paSjL/build/release/bin/postgresql.jar:/tmp/dist-test-task7paSjL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37507/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-hwlj.c.gcp-upstream.internal';"
2026-09-26 20:49:53,564 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 20:49:53,564 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7paSjL/build/release/bin/postgresql.jar:/tmp/dist-test-task7paSjL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37507/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 20:49:53,816 [I] DB_PATCHES have already been applied
I20260926 20:49:53.822082 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455725523895-392-0/minicluster-data/ranger-admin:/tmp/dist-test-task7paSjL/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task7paSjL/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task7paSjL/build/release/bin/postgresql.jar:/tmp/dist-test-task7paSjL/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 20:49:53.822147 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260926 20:49:53.825268 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:59639
Sep 26, 2026 8:49:54 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 8:49:54 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 8:49:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455725523895-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 8:49:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455725523895-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 8:49:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455725523895-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 8:49:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455725523895-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 8:49:54 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 20:49:54.976233 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.533s user 0.000s sys 0.009s
Sep 26, 2026 8:50:14 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 8:50:14 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 26, 2026 8:50:14 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 8:50:14 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 8:50:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 8:50:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 8:50:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 8:50:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 8:50:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 8:50:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 8:50:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 8:50:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 8:50:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 8:50:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 8:50:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 8:50:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 8:50:14 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 8:50:15 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 20:50:15.906929 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455725523895-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455725523895-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455725523895-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455725523895-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 20:50:15 dist-test-slave-hwlj krb5kdc[11119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455815, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 20:50:16.003937 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 20:50:16.014168 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 20:50:16.962762 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 20:50:16.962862 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455725523895-392-0/minicluster-data/ranger-kms
I20260926 20:50:16.962886 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:37507
2026-09-26 20:50:17,145 [I] DB FLAVOR :POSTGRES
2026-09-26 20:50:17,146 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 20:50:17,146 [I] Checking connection
2026-09-26 20:50:17,146 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7paSjL/build/release/bin/postgresql.jar:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455725523895-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37507/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 20:50:17,401 [I] connection success
2026-09-26 20:50:17,401 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 20:50:17,402 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 20:50:17,402 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7paSjL/build/release/bin/postgresql.jar:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455725523895-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37507/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 20:50:17,641 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 20:50:17,641 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 20:50:17,641 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 20:50:17,641 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7paSjL/build/release/bin/postgresql.jar:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455725523895-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37507/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455725523895-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 20:50:17,900 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 20:50:18.361635 392 mini_ranger_kms.cc:310] Created kms service
I20260926 20:50:18.535075 392 mini_ranger_kms.cc:326] Created kudu user
I20260926 20:50:18.581239 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 20:50:18 dist-test-slave-hwlj krb5kdc[11119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455815, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 20:50:18.994453 392 mini_ranger_kms.cc:384] Added ranger policy
I20260926 20:50:18.994570 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455725523895-392-0/minicluster-data/ranger-kms:/tmp/dist-test-task7paSjL/build/release/bin/postgresql.jar:/tmp/dist-test-task7paSjL/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task7paSjL/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 20:50:18.994592 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260926 20:50:18.997870 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11512
I20260926 20:50:18.998023 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:36127
Sep 26, 2026 8:50:20 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 8:50:20 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 8:50:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455725523895-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 8:50:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455725523895-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 8:50:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455725523895-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 8:50:20 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 26 20:50:21 dist-test-slave-hwlj krb5kdc[11119](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790455821, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 20:50:21 dist-test-slave-hwlj krb5kdc[11119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455821, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 20:50:22 dist-test-slave-hwlj krb5kdc[11119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455821, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 20:50:52 dist-test-slave-hwlj krb5kdc[11119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455821, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 20:50:52 dist-test-slave-hwlj krb5kdc[11119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455821, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 20:51:22 dist-test-slave-hwlj krb5kdc[11119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455821, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 20:51:22 dist-test-slave-hwlj krb5kdc[11119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455821, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 20:51:52 dist-test-slave-hwlj krb5kdc[11119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455821, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 20:51:52 dist-test-slave-hwlj krb5kdc[11119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455821, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 20:52:19.509657 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 123.506s user 0.035s sys 0.259s
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:1013: Failure
Failed
Bad status: Timed out: Failed to start the Ranger KMS service: process with PID 11512 is not yet bound to any port at the specified addresses; last attempt running lsof returned 'Not found: could not find pattern of a bound port in lsof output: p11512
f196
n127.0.0.1:53588->127.0.98.20:37507'
I20260926 20:52:19.510452 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 20:52:19.531085 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 20:52:19.531199 392 mini_ranger.cc:66] Stopping Ranger...
I20260926 20:52:19.876362 392 mini_ranger.cc:68] Stopped Ranger
2026-09-26 20:52:19.876 UTC [11149] LOG: received smart shutdown request
2026-09-26 20:52:19.880 UTC [11149] LOG: background worker "logical replication launcher" (PID 11158) exited with exit code 1
2026-09-26 20:52:19.880 UTC [11153] LOG: shutting down
2026-09-26 20:52:20.047 UTC [11149] LOG: database system is shut down
2026-09-26T20:52:20Z chronyd exiting
I20260926 20:52:20.062536 392 test_util.cc:175] -----------------------------------------------
I20260926 20:52:20.062589 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455725523895-392-0
[ FAILED ] SecurityITest.TestEncryptionWithKMSIntegration (158431 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:52:20 dist-test-slave-hwlj krb5kdc[11647](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:52:20 dist-test-slave-hwlj krb5kdc[11647](info): set up 2 sockets
Sep 26 20:52:20 dist-test-slave-hwlj krb5kdc[11647](info): commencing operation
krb5kdc: starting...
W20260926 20:52:21.091266 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/test-user.keytab.
Sep 26 20:52:21 dist-test-slave-hwlj krb5kdc[11647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:52:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:52:21Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260926 20:52:21.259862 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task7paSjL/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/postgres -l logfile start
2026-09-26 20:52:23.900 UTC [11677] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 20:52:23.900 UTC [11677] LOG: listening on IPv4 address "127.0.98.20", port 43359
2026-09-26 20:52:23.904 UTC [11677] LOG: listening on Unix socket "/tmp/.s.PGSQL.43359"
2026-09-26 20:52:23.926 UTC [11680] LOG: database system was shut down at 2026-09-26 20:52:22 UTC
2026-09-26 20:52:23.930 UTC [11677] LOG: database system is ready to accept connections
I20260926 20:52:24.879068 392 mini_postgres.cc:96] Postgres bound to 43359
2026-09-26 20:52:24.885 UTC [11688] FATAL: database "slave" does not exist
127.0.98.20:43359 - accepting connections
I20260926 20:52:24.885835 392 mini_ranger.cc:161] Starting Ranger...
I20260926 20:52:24.900664 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 20:52:25.434674 392 mini_ranger.cc:87] Created ranger Postgres database
I20260926 20:52:25.434762 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ranger-admin
2026-09-26 20:52:25,713 [I] DB FLAVOR :POSTGRES
2026-09-26 20:52:25,713 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 20:52:25,713 [I] Checking connection..
2026-09-26 20:52:25,713 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7paSjL/build/release/bin/postgresql.jar:/tmp/dist-test-task7paSjL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 20:52:25,944 [I] Checking connection passed.
2026-09-26 20:52:25,944 [I] --------- Verifying version history table ---------
2026-09-26 20:52:25,944 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7paSjL/build/release/bin/postgresql.jar:/tmp/dist-test-task7paSjL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 20:52:26,173 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 20:52:26,174 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7paSjL/build/release/bin/postgresql.jar:/tmp/dist-test-task7paSjL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 20:52:26,425 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 20:52:26,425 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 20:52:26,425 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7paSjL/build/release/bin/postgresql.jar:/tmp/dist-test-task7paSjL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43359/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7paSjL/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 20:52:26.672 UTC [11752] WARNING: there is no transaction in progress
2026-09-26 20:52:26,679 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 20:52:26,679 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7paSjL/build/release/bin/postgresql.jar:/tmp/dist-test-task7paSjL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 20:52:26,920 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 20:52:26,920 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 20:52:26,920 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7paSjL/build/release/bin/postgresql.jar:/tmp/dist-test-task7paSjL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 20:52:27,153 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7paSjL/build/release/bin/postgresql.jar:/tmp/dist-test-task7paSjL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 20:52:27,394 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7paSjL/build/release/bin/postgresql.jar:/tmp/dist-test-task7paSjL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-hwlj.c.gcp-upstream.internal','N') ;"
2026-09-26 20:52:27,648 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7paSjL/build/release/bin/postgresql.jar:/tmp/dist-test-task7paSjL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 20:52:27,892 [I] Table x_portal_user does not exist in database ranger
2026-09-26 20:52:27,893 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7paSjL/build/release/bin/postgresql.jar:/tmp/dist-test-task7paSjL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 20:52:28,126 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 20:52:28,127 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7paSjL/build/release/bin/postgresql.jar:/tmp/dist-test-task7paSjL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 20:52:28,376 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7paSjL/build/release/bin/postgresql.jar:/tmp/dist-test-task7paSjL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 20:52:28,629 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 20:52:28,629 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7paSjL/build/release/bin/postgresql.jar:/tmp/dist-test-task7paSjL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43359/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7paSjL/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 20:52:29.637 UTC [11860] WARNING: there is no transaction in progress
2026-09-26 20:52:29.653 UTC [11860] WARNING: there is no transaction in progress
2026-09-26 20:52:29.670 UTC [11860] WARNING: there is no transaction in progress
2026-09-26 20:52:29.685 UTC [11860] WARNING: there is no transaction in progress
2026-09-26 20:52:29.702 UTC [11860] WARNING: there is no transaction in progress
2026-09-26 20:52:29.720 UTC [11860] WARNING: there is no transaction in progress
2026-09-26 20:52:29.777 UTC [11860] WARNING: there is no transaction in progress
2026-09-26 20:52:29.784 UTC [11860] WARNING: there is no transaction in progress
2026-09-26 20:52:29.798 UTC [11860] WARNING: there is no transaction in progress
2026-09-26 20:52:29.808 UTC [11860] WARNING: there is no transaction in progress
2026-09-26 20:52:29.820 UTC [11860] WARNING: there is no transaction in progress
2026-09-26 20:52:29.835 UTC [11860] WARNING: there is no transaction in progress
2026-09-26 20:52:29.846 UTC [11860] WARNING: there is no transaction in progress
2026-09-26 20:52:29.854 UTC [11860] WARNING: there is no transaction in progress
2026-09-26 20:52:29.861 UTC [11860] WARNING: there is no transaction in progress
2026-09-26 20:52:29.869 UTC [11860] WARNING: there is no transaction in progress
2026-09-26 20:52:30,533 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 20:52:30,533 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7paSjL/build/release/bin/postgresql.jar:/tmp/dist-test-task7paSjL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 20:52:30,756 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 20:52:30,756 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7paSjL/build/release/bin/postgresql.jar:/tmp/dist-test-task7paSjL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 20:52:31,001 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 20:52:31,001 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7paSjL/build/release/bin/postgresql.jar:/tmp/dist-test-task7paSjL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 20:52:31,264 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 20:52:31,265 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7paSjL/build/release/bin/postgresql.jar:/tmp/dist-test-task7paSjL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 20:52:31,522 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7paSjL/build/release/bin/postgresql.jar:/tmp/dist-test-task7paSjL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 20:52:31,760 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7paSjL/build/release/bin/postgresql.jar:/tmp/dist-test-task7paSjL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-hwlj.c.gcp-upstream.internal';"
2026-09-26 20:52:32,002 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 20:52:32,002 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7paSjL/build/release/bin/postgresql.jar:/tmp/dist-test-task7paSjL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 20:52:32,255 [I] DB_PATCHES have already been applied
I20260926 20:52:32.261174 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ranger-admin:/tmp/dist-test-task7paSjL/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task7paSjL/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task7paSjL/build/release/bin/postgresql.jar:/tmp/dist-test-task7paSjL/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 20:52:32.261243 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260926 20:52:32.264675 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:54599
Sep 26, 2026 8:52:32 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 8:52:32 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 8:52:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 8:52:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 8:52:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 8:52:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 8:52:33 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 20:52:33.448966 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.563s user 0.001s sys 0.007s
Sep 26, 2026 8:52:54 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 8:52:54 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 8:52:54 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 8:52:54 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 8:52:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 8:52:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 8:52:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 8:52:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 8:52:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 8:52:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 8:52:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 8:52:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 8:52:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 8:52:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 8:52:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 8:52:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 8:52:54 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 8:52:55 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 20:52:55.749332 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 20:52:55 dist-test-slave-hwlj krb5kdc[11647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455975, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 20:52:55.848609 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 20:52:55.858867 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 20:52:56.790987 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 20:52:56.791074 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ranger-kms
I20260926 20:52:56.791098 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:43359
2026-09-26 20:52:56,957 [I] DB FLAVOR :POSTGRES
2026-09-26 20:52:56,957 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 20:52:56,957 [I] Checking connection
2026-09-26 20:52:56,957 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7paSjL/build/release/bin/postgresql.jar:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43359/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 20:52:57,205 [I] connection success
2026-09-26 20:52:57,205 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 20:52:57,205 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 20:52:57,205 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7paSjL/build/release/bin/postgresql.jar:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43359/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 20:52:57,473 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 20:52:57,473 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 20:52:57,473 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 20:52:57,474 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7paSjL/build/release/bin/postgresql.jar:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43359/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 20:52:57,726 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 20:52:58.243096 392 mini_ranger_kms.cc:310] Created kms service
I20260926 20:52:58.475971 392 mini_ranger_kms.cc:326] Created kudu user
I20260926 20:52:58.520222 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 20:52:58 dist-test-slave-hwlj krb5kdc[11647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455975, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 20:52:58.930224 392 mini_ranger_kms.cc:384] Added ranger policy
I20260926 20:52:58.930356 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ranger-kms:/tmp/dist-test-task7paSjL/build/release/bin/postgresql.jar:/tmp/dist-test-task7paSjL/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task7paSjL/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 20:52:58.930377 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260926 20:52:58.933694 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12040
I20260926 20:52:58.933782 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:34243
Sep 26, 2026 8:53:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 8:53:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 8:53:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 8:53:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 8:53:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 8:53:00 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 20:53:01.181783 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.333s user 0.000s sys 0.148s
I20260926 20:53:01.181888 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 20:53:01.181974 392 mini_ranger_kms.cc:401] 127.0.98.20:34243/kms/v1/keys
Sep 26 20:53:01 dist-test-slave-hwlj krb5kdc[11647](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790455981, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 20:53:01 dist-test-slave-hwlj krb5kdc[11647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455981, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 20:53:02 dist-test-slave-hwlj krb5kdc[11647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455981, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 20:53:04 dist-test-slave-hwlj krb5kdc[11647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 20:53:04.408030 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33927
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33927
--ranger_config_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34243/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:53:04.490908 12091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:53:04.491111 12091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:53:04.491132 12091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:53:04.492349 12091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:53:04.492393 12091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:53:04.492408 12091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:53:04.492420 12091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:53:04.492461 12091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:53:04.494015 12091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:34243/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33927
--ranger_config_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:53:04.494253 12091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:53:04.494481 12091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:53:04.499231 12096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:04.499226 12097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:04.499221 12099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:04.499656 12091 server_base.cc:1034] running on GCE node
I20260926 20:53:04.499922 12091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:53:04.500218 12091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:53:04.501376 12091 hybrid_clock.cc:648] HybridClock initialized: now 1790455984501367 us; error 37 us; skew 500 ppm
Sep 26 20:53:04 dist-test-slave-hwlj krb5kdc[11647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455984, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:53:04.508008 12091 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:53:04.508525 12091 webserver.cc:460] Webserver started at http://127.0.98.62:37105/ using document root <none> and password file <none>
I20260926 20:53:04.508752 12091 fs_manager.cc:359] Metadata directory not provided
I20260926 20:53:04.508810 12091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:53:04.508970 12091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 20:53:04 dist-test-slave-hwlj krb5kdc[11647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455984, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 20:53:04.566470 12091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/master-0/data/instance:
uuid: "8f2e42251ae2400fa105f2dfc79d506c"
format_stamp: "Formatted at 2026-09-26 20:53:04 on dist-test-slave-hwlj"
server_key: "81031e1e8b008fc0e602f315523b8a91"
server_key_iv: "9a6e10f2880866778eacce3f5eda4a0a"
server_key_version: "kuduclusterkey@0"
I20260926 20:53:04.566879 12091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/master-0/wal/instance:
uuid: "8f2e42251ae2400fa105f2dfc79d506c"
format_stamp: "Formatted at 2026-09-26 20:53:04 on dist-test-slave-hwlj"
server_key: "81031e1e8b008fc0e602f315523b8a91"
server_key_iv: "9a6e10f2880866778eacce3f5eda4a0a"
server_key_version: "kuduclusterkey@0"
I20260926 20:53:04.568357 12091 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260926 20:53:04.582904 12106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:04.583088 12091 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:53:04.583231 12091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "8f2e42251ae2400fa105f2dfc79d506c"
format_stamp: "Formatted at 2026-09-26 20:53:04 on dist-test-slave-hwlj"
server_key: "81031e1e8b008fc0e602f315523b8a91"
server_key_iv: "9a6e10f2880866778eacce3f5eda4a0a"
server_key_version: "kuduclusterkey@0"
I20260926 20:53:04.583356 12091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:53:04.610847 12091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:53:04.612918 12091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:53:04.613076 12091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:53:04.617571 12091 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33927
I20260926 20:53:04.617614 12158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33927 every 8 connection(s)
I20260926 20:53:04.617959 12091 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:53:04.618585 12159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:53:04.620844 12159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f2e42251ae2400fa105f2dfc79d506c: Bootstrap starting.
I20260926 20:53:04.621439 12159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f2e42251ae2400fa105f2dfc79d506c: Neither blocks nor log segments found. Creating new log.
I20260926 20:53:04.621724 12159 log.cc:824] T 00000000000000000000000000000000 P 8f2e42251ae2400fa105f2dfc79d506c: Log is configured to *not* fsync() on all Append() calls
I20260926 20:53:04.622442 12159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f2e42251ae2400fa105f2dfc79d506c: No bootstrap required, opened a new log
I20260926 20:53:04.624394 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 12091
I20260926 20:53:04.624504 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/master-0/wal/instance
I20260926 20:53:04.624711 392 external_mini_cluster.cc:1428] Setting key ab293434a12aa5eacc28d93f7811a0bb
I20260926 20:53:04.624738 12159 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f2e42251ae2400fa105f2dfc79d506c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f2e42251ae2400fa105f2dfc79d506c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33927 } }
I20260926 20:53:04.624931 12159 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f2e42251ae2400fa105f2dfc79d506c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:53:04.624989 12159 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f2e42251ae2400fa105f2dfc79d506c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2e42251ae2400fa105f2dfc79d506c, State: Initialized, Role: FOLLOWER
I20260926 20:53:04.625136 12159 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f2e42251ae2400fa105f2dfc79d506c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f2e42251ae2400fa105f2dfc79d506c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33927 } }
I20260926 20:53:04.625273 12159 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f2e42251ae2400fa105f2dfc79d506c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:53:04.625329 12159 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f2e42251ae2400fa105f2dfc79d506c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:53:04.625443 12159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f2e42251ae2400fa105f2dfc79d506c [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:53:04.625991 12159 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f2e42251ae2400fa105f2dfc79d506c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f2e42251ae2400fa105f2dfc79d506c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33927 } }
I20260926 20:53:04.626108 12159 leader_election.cc:302] T 00000000000000000000000000000000 P 8f2e42251ae2400fa105f2dfc79d506c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f2e42251ae2400fa105f2dfc79d506c; no voters:
I20260926 20:53:04.626276 12159 leader_election.cc:288] T 00000000000000000000000000000000 P 8f2e42251ae2400fa105f2dfc79d506c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:53:04.626327 12164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f2e42251ae2400fa105f2dfc79d506c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:53:04.626528 12159 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f2e42251ae2400fa105f2dfc79d506c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:53:04.626539 12164 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f2e42251ae2400fa105f2dfc79d506c [term 1 LEADER]: Becoming Leader. State: Replica: 8f2e42251ae2400fa105f2dfc79d506c, State: Running, Role: LEADER
I20260926 20:53:04.627887 12164 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f2e42251ae2400fa105f2dfc79d506c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f2e42251ae2400fa105f2dfc79d506c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33927 } }
I20260926 20:53:04.628561 12159 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 20:53:04.628585 12165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f2e42251ae2400fa105f2dfc79d506c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f2e42251ae2400fa105f2dfc79d506c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f2e42251ae2400fa105f2dfc79d506c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33927 } } }
I20260926 20:53:04.628594 12166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f2e42251ae2400fa105f2dfc79d506c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f2e42251ae2400fa105f2dfc79d506c. Latest consensus state: current_term: 1 leader_uuid: "8f2e42251ae2400fa105f2dfc79d506c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f2e42251ae2400fa105f2dfc79d506c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33927 } } }
I20260926 20:53:04.628660 12165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f2e42251ae2400fa105f2dfc79d506c [sys.catalog]: This master's current role is: LEADER
I20260926 20:53:04.628706 12166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f2e42251ae2400fa105f2dfc79d506c [sys.catalog]: This master's current role is: LEADER
I20260926 20:53:04.631551 12179 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:53:04.632848 12179 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 20:53:04 dist-test-slave-hwlj krb5kdc[11647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:53:04.634872 12179 catalog_manager.cc:1347] Generated new cluster ID: fbb492a8c1214558beab2c6d947bd2ca
I20260926 20:53:04.634917 12179 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:53:04.640115 12163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:04.625246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37232 (local address 127.0.98.62:33927)
0926 20:53:04.625421 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:04.625433 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:04.625476 (+ 43us) server_negotiation.cc:407] Connection header received
0926 20:53:04.625644 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:04.625650 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:04.625694 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:04.625852 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:04.626620 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:04.627652 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:04.628779 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:04.631940 (+ 3161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:04.632158 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:04.635124 (+ 2966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:04.635140 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:04.635141 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:04.635152 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:04.637292 (+ 2140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:04.637761 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:04.637765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:04.637765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:04.637901 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:04.638141 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:04.638142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:04.638143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:04.638494 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:04.639534 (+ 1040us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:04.639626 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 20:53:04.639842 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":90,"threads_started":1}
I20260926 20:53:04.650386 12179 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:53:04.651182 12179 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:53:04.653653 12179 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f2e42251ae2400fa105f2dfc79d506c: Generated new TSK 0
I20260926 20:53:04.653833 12179 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 20:53:05.363162 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34243/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33927
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:53:05.464423 12204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:53:05.464679 12204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:53:05.464722 12204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:53:05.466127 12204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:53:05.466261 12204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:53:05.466347 12204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:53:05.468089 12204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:34243/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:53:05.468390 12204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:53:05.468657 12204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:53:05.474340 12212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:05.474347 12209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:05.474609 12204 server_base.cc:1034] running on GCE node
W20260926 20:53:05.474858 12210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:05.475106 12204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:53:05.475440 12204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:53:05.476622 12204 hybrid_clock.cc:648] HybridClock initialized: now 1790455985476609 us; error 30 us; skew 500 ppm
Sep 26 20:53:05 dist-test-slave-hwlj krb5kdc[11647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:53:05.479885 12204 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 20:53:05.480379 12204 webserver.cc:460] Webserver started at http://127.0.98.1:39245/ using document root <none> and password file <none>
I20260926 20:53:05.480626 12204 fs_manager.cc:359] Metadata directory not provided
I20260926 20:53:05.480692 12204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:53:05.480849 12204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 20:53:05 dist-test-slave-hwlj krb5kdc[11647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 20:53:05.496058 12204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-0/data/instance:
uuid: "e1969346889647daa3791a84afa2f6d4"
format_stamp: "Formatted at 2026-09-26 20:53:05 on dist-test-slave-hwlj"
server_key: "3f9468ddbd192730a7b5e0fe324d0ef8"
server_key_iv: "6e08100449372b4d14277d015b4cd313"
server_key_version: "kuduclusterkey@0"
I20260926 20:53:05.496894 12204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e1969346889647daa3791a84afa2f6d4"
format_stamp: "Formatted at 2026-09-26 20:53:05 on dist-test-slave-hwlj"
server_key: "3f9468ddbd192730a7b5e0fe324d0ef8"
server_key_iv: "6e08100449372b4d14277d015b4cd313"
server_key_version: "kuduclusterkey@0"
I20260926 20:53:05.498927 12204 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260926 20:53:05.513219 12219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:05.513440 12204 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:53:05.513530 12204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-0/wal
uuid: "e1969346889647daa3791a84afa2f6d4"
format_stamp: "Formatted at 2026-09-26 20:53:05 on dist-test-slave-hwlj"
server_key: "3f9468ddbd192730a7b5e0fe324d0ef8"
server_key_iv: "6e08100449372b4d14277d015b4cd313"
server_key_version: "kuduclusterkey@0"
I20260926 20:53:05.513630 12204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:53:05.560006 12204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:53:05.562520 12204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:53:05.562686 12204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:53:05.562997 12204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:53:05.563421 12204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:53:05.563474 12204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:05.563531 12204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:53:05.563557 12204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:05.571466 12204 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36809
I20260926 20:53:05.571540 12334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36809 every 8 connection(s)
I20260926 20:53:05.571971 12204 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:53:05.576100 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 12204
I20260926 20:53:05.576195 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-0/wal/instance
I20260926 20:53:05.576416 392 external_mini_cluster.cc:1428] Setting key 15be42f797330d1a8d9fcad4186724d2
Sep 26 20:53:05 dist-test-slave-hwlj krb5kdc[11647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:53:05.582438 12338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:05.572781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45297 (local address 127.0.98.62:33927)
0926 20:53:05.572922 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:05.572924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:05.573133 (+ 209us) server_negotiation.cc:407] Connection header received
0926 20:53:05.573666 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:05.573668 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:05.573682 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:05.573748 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:05.574373 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:05.574891 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:05.576004 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:05.576327 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:05.576399 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:05.579195 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:05.579197 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:05.579197 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:05.579205 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:05.581011 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:05.581435 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:05.581437 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:05.581437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:05.581503 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:05.581763 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:05.581764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:05.581764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:05.581887 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:05.581962 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:05.582252 (+ 290us) server_negotiation.cc:299] Negotiation successful
0926 20:53:05.582338 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":83,"threads_started":1}
I20260926 20:53:05.582639 12337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:05.572771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33927 (local address 127.0.98.1:45297)
0926 20:53:05.572994 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:05.573035 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:05.573554 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:05.573821 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:05.573825 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:05.574020 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:05.574284 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:05.574289 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:05.574987 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:05.574988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:05.575870 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:05.575872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:05.576490 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:05.576491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:05.576703 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:05.577343 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:05.577355 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:05.579047 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:53:05.581117 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:05.581124 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:05.581126 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:05.581357 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:05.581569 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:05.581570 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:05.581571 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:05.581685 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:05.581951 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:53:05.581958 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:53:05.582115 (+ 157us) client_negotiation.cc:769] Sending connection context
0926 20:53:05.582229 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 20:53:05.582383 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":84,"threads_started":1}
I20260926 20:53:05.583178 12335 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33927
I20260926 20:53:05.583280 12335 heartbeater.cc:464] Registering TS with master...
I20260926 20:53:05.583461 12335 heartbeater.cc:511] Master 127.0.98.62:33927 requested a full tablet report, sending...
I20260926 20:53:05.583974 12123 ts_manager.cc:194] Registered new tserver with Master: e1969346889647daa3791a84afa2f6d4 (127.0.98.1:36809)
I20260926 20:53:05.584575 12123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45297
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 20:53:05.610167 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34243/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33927
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:53:05 dist-test-slave-hwlj krb5kdc[11647](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790455985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 20:53:05.736094 12341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:53:05.736496 12341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:53:05.736584 12341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:53:05.738557 12341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:53:05.738641 12341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:53:05.738686 12341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:53:05.741087 12341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:34243/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:53:05.741582 12341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:53:05.741914 12341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:53:05.749079 12351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:05.749500 12348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:05.750486 12341 server_base.cc:1034] running on GCE node
W20260926 20:53:05.751911 12349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:05.752177 12341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:53:05.752489 12341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:53:05.753669 12341 hybrid_clock.cc:648] HybridClock initialized: now 1790455985753643 us; error 44 us; skew 500 ppm
Sep 26 20:53:05 dist-test-slave-hwlj krb5kdc[11647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:53:05.757190 12341 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 20:53:05.757772 12341 webserver.cc:460] Webserver started at http://127.0.98.2:41603/ using document root <none> and password file <none>
I20260926 20:53:05.758046 12341 fs_manager.cc:359] Metadata directory not provided
I20260926 20:53:05.758107 12341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:53:05.758315 12341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 20:53:05 dist-test-slave-hwlj krb5kdc[11647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 20:53:05.772832 12341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-1/data/instance:
uuid: "1d3a0e73ec5945fdb930973744c5f039"
format_stamp: "Formatted at 2026-09-26 20:53:05 on dist-test-slave-hwlj"
server_key: "7bb54c127b364a4c24d0ba0faa024978"
server_key_iv: "ec8a64fb3edf06a8a31af457ed39e6ae"
server_key_version: "kuduclusterkey@0"
I20260926 20:53:05.773316 12341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1d3a0e73ec5945fdb930973744c5f039"
format_stamp: "Formatted at 2026-09-26 20:53:05 on dist-test-slave-hwlj"
server_key: "7bb54c127b364a4c24d0ba0faa024978"
server_key_iv: "ec8a64fb3edf06a8a31af457ed39e6ae"
server_key_version: "kuduclusterkey@0"
I20260926 20:53:05.779994 12341 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.000s
I20260926 20:53:05.793850 12359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:05.794095 12341 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:53:05.794202 12341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-1/wal
uuid: "1d3a0e73ec5945fdb930973744c5f039"
format_stamp: "Formatted at 2026-09-26 20:53:05 on dist-test-slave-hwlj"
server_key: "7bb54c127b364a4c24d0ba0faa024978"
server_key_iv: "ec8a64fb3edf06a8a31af457ed39e6ae"
server_key_version: "kuduclusterkey@0"
I20260926 20:53:05.794325 12341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:53:05.818800 12341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:53:05.821357 12341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:53:05.821513 12341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:53:05.821767 12341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:53:05.822139 12341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:53:05.822190 12341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:05.822237 12341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:53:05.822260 12341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:05.829370 12341 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42389
I20260926 20:53:05.829964 12341 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
I20260926 20:53:05.832964 12472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42389 every 8 connection(s)
I20260926 20:53:05.838717 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 12341
I20260926 20:53:05.838806 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-1/wal/instance
I20260926 20:53:05.839005 392 external_mini_cluster.cc:1428] Setting key 519f6638511c60660efa902580286352
Sep 26 20:53:05 dist-test-slave-hwlj krb5kdc[11647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:53:05.845218 12475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:05.833583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33927 (local address 127.0.98.2:39947)
0926 20:53:05.833768 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:05.833811 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:05.834567 (+ 756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:05.834826 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:05.834829 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:05.835086 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:05.835374 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:05.835381 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:05.836477 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:05.836478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:05.837922 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:05.837926 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:05.838523 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:05.838525 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:05.838683 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:05.839434 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:05.839454 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:05.841254 (+ 1800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:53:05.843386 (+ 2132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:05.843394 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:05.843397 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:05.843731 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:05.843994 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:05.843995 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:05.843995 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:05.844150 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:05.844470 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:53:05.844477 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:53:05.844673 (+ 196us) client_negotiation.cc:769] Sending connection context
0926 20:53:05.844811 (+ 138us) client_negotiation.cc:240] Negotiation successful
0926 20:53:05.844977 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":124,"threads_started":1}
I20260926 20:53:05.845973 12338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:05.833599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39947 (local address 127.0.98.62:33927)
0926 20:53:05.833857 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:05.833860 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:05.833892 (+ 32us) server_negotiation.cc:407] Connection header received
0926 20:53:05.834672 (+ 780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:05.834674 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:05.834686 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:05.834750 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:05.835464 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:05.836194 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:05.838063 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:05.838389 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:05.838470 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:05.841456 (+ 2986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:05.841458 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:05.841458 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:05.841479 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:05.843290 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:05.843844 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:05.843845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:05.843846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:05.843910 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:05.844234 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:05.844235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:05.844235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:05.844380 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:05.844443 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:05.845784 (+ 1341us) server_negotiation.cc:299] Negotiation successful
0926 20:53:05.845881 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233}
I20260926 20:53:05.846601 12473 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33927
I20260926 20:53:05.846697 12473 heartbeater.cc:464] Registering TS with master...
I20260926 20:53:05.846912 12473 heartbeater.cc:511] Master 127.0.98.62:33927 requested a full tablet report, sending...
I20260926 20:53:05.847290 12123 ts_manager.cc:194] Registered new tserver with Master: 1d3a0e73ec5945fdb930973744c5f039 (127.0.98.2:42389)
I20260926 20:53:05.847949 12123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39947
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 20:53:05.880573 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34243/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33927
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:53:06.009128 12478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:53:06.009502 12478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:53:06.009595 12478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:53:06.011078 12478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:53:06.011166 12478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:53:06.011219 12478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:53:06.012979 12478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:34243/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:53:06.013264 12478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:53:06.013509 12478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:53:06.021720 12484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:06.021720 12483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:06.022190 12486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:06.022774 12478 server_base.cc:1034] running on GCE node
I20260926 20:53:06.023030 12478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:53:06.023348 12478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:53:06.024533 12478 hybrid_clock.cc:648] HybridClock initialized: now 1790455986024509 us; error 38 us; skew 500 ppm
Sep 26 20:53:06 dist-test-slave-hwlj krb5kdc[11647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:53:06.028112 12478 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 20:53:06.028720 12478 webserver.cc:460] Webserver started at http://127.0.98.3:37675/ using document root <none> and password file <none>
I20260926 20:53:06.029038 12478 fs_manager.cc:359] Metadata directory not provided
I20260926 20:53:06.029109 12478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:53:06.029268 12478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 20:53:06 dist-test-slave-hwlj krb5kdc[11647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 20:53:06.049981 12478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-2/data/instance:
uuid: "b215d4f3878b4a818486c492098002f3"
format_stamp: "Formatted at 2026-09-26 20:53:06 on dist-test-slave-hwlj"
server_key: "c842473da7caa967a61ad39c369d6210"
server_key_iv: "97d43eae909f21f7216c54d0a43cbdaf"
server_key_version: "kuduclusterkey@0"
I20260926 20:53:06.050613 12478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b215d4f3878b4a818486c492098002f3"
format_stamp: "Formatted at 2026-09-26 20:53:06 on dist-test-slave-hwlj"
server_key: "c842473da7caa967a61ad39c369d6210"
server_key_iv: "97d43eae909f21f7216c54d0a43cbdaf"
server_key_version: "kuduclusterkey@0"
I20260926 20:53:06.052747 12478 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:53:06.067406 12493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:06.067585 12478 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 20:53:06.067677 12478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-2/wal
uuid: "b215d4f3878b4a818486c492098002f3"
format_stamp: "Formatted at 2026-09-26 20:53:06 on dist-test-slave-hwlj"
server_key: "c842473da7caa967a61ad39c369d6210"
server_key_iv: "97d43eae909f21f7216c54d0a43cbdaf"
server_key_version: "kuduclusterkey@0"
I20260926 20:53:06.067828 12478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:53:06.090167 12478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:53:06.094358 12478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:53:06.094515 12478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:53:06.094786 12478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:53:06.095199 12478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:53:06.095265 12478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:06.095307 12478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:53:06.095337 12478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:06.103566 12478 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37203
I20260926 20:53:06.103631 12606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37203 every 8 connection(s)
I20260926 20:53:06.104087 12478 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
I20260926 20:53:06.111822 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 12478
I20260926 20:53:06.111933 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-2/wal/instance
I20260926 20:53:06.112126 392 external_mini_cluster.cc:1428] Setting key e2686d178de0834d8c30f9b61cb7483a
Sep 26 20:53:06 dist-test-slave-hwlj krb5kdc[11647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:53:06.123961 12338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:06.105063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52887 (local address 127.0.98.62:33927)
0926 20:53:06.105121 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:06.105123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:06.108072 (+ 2949us) server_negotiation.cc:407] Connection header received
0926 20:53:06.108845 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:06.108848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:06.108861 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:06.108950 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:06.109743 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:06.110579 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:06.117259 (+ 6680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:06.117574 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:06.117618 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:06.120382 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:06.120384 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:06.120384 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:06.120392 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:06.122173 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:06.122702 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:06.122703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:06.122704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:06.122774 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:06.123103 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:06.123105 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:06.123105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:06.123255 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:06.123329 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:06.123736 (+ 407us) server_negotiation.cc:299] Negotiation successful
0926 20:53:06.123858 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 20:53:06.124533 12609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:06.104985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33927 (local address 127.0.98.3:52887)
0926 20:53:06.107893 (+ 2908us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:06.107955 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:06.108709 (+ 754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:06.109039 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:06.109044 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:06.109332 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:06.109629 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:06.109636 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:06.115383 (+ 5747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:06.115385 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:06.117144 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:06.117148 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:06.117737 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:06.117738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:06.117883 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:06.118524 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:06.118534 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:06.120279 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:53:06.122275 (+ 1996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:06.122283 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:06.122285 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:06.122604 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:06.122844 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:06.122845 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:06.122846 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:06.123007 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:06.123366 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:53:06.123373 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:53:06.123559 (+ 186us) client_negotiation.cc:769] Sending connection context
0926 20:53:06.124104 (+ 545us) client_negotiation.cc:240] Negotiation successful
0926 20:53:06.124289 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2827,"thread_start_us":88,"threads_started":1}
I20260926 20:53:06.124993 12607 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33927
I20260926 20:53:06.125104 12607 heartbeater.cc:464] Registering TS with master...
I20260926 20:53:06.125324 12607 heartbeater.cc:511] Master 127.0.98.62:33927 requested a full tablet report, sending...
I20260926 20:53:06.125694 12123 ts_manager.cc:194] Registered new tserver with Master: b215d4f3878b4a818486c492098002f3 (127.0.98.3:37203)
I20260926 20:53:06.126190 12123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52887
I20260926 20:53:06.127202 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:53:06.127964 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 12204
I20260926 20:53:06.133584 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 12341
I20260926 20:53:06.139209 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 12478
I20260926 20:53:06.143468 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 12091
I20260926 20:53:06.149045 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33927
--webserver_interface=127.0.98.62
--webserver_port=37105
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33927
--ranger_config_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34243/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:34243/kms with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:53:06.230899 12610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:53:06.231133 12610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:53:06.231155 12610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:53:06.232370 12610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:53:06.232417 12610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:53:06.232431 12610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:53:06.232445 12610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:53:06.232456 12610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:53:06.234176 12610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:34243/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33927
--ranger_config_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:53:06.234436 12610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:53:06.234670 12610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:53:06.239847 12610 server_base.cc:1034] running on GCE node
W20260926 20:53:06.239796 12615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:06.239831 12616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:06.239816 12618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:06.240278 12610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:53:06.240590 12610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:53:06.241770 12610 hybrid_clock.cc:648] HybridClock initialized: now 1790455986241761 us; error 34 us; skew 500 ppm
Sep 26 20:53:06 dist-test-slave-hwlj krb5kdc[11647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:53:06.244524 12610 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:53:06.245013 12610 webserver.cc:460] Webserver started at http://127.0.98.62:37105/ using document root <none> and password file <none>
I20260926 20:53:06.245234 12610 fs_manager.cc:359] Metadata directory not provided
I20260926 20:53:06.245308 12610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 20:53:06 dist-test-slave-hwlj krb5kdc[11647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 20:53:06.294070 12610 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:53:06.294886 12626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:06.295082 12610 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:53:06.295184 12610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "8f2e42251ae2400fa105f2dfc79d506c"
format_stamp: "Formatted at 2026-09-26 20:53:04 on dist-test-slave-hwlj"
server_key: "81031e1e8b008fc0e602f315523b8a91"
server_key_iv: "9a6e10f2880866778eacce3f5eda4a0a"
server_key_version: "kuduclusterkey@0"
I20260926 20:53:06.295637 12610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:53:06.305779 12610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:53:06.307586 12610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:53:06.307701 12610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:53:06.311851 12610 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33927
I20260926 20:53:06.311899 12678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33927 every 8 connection(s)
I20260926 20:53:06.312270 12610 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:53:06.313566 12679 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:53:06.314353 12679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f2e42251ae2400fa105f2dfc79d506c: Bootstrap starting.
I20260926 20:53:06.314436 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 12610
I20260926 20:53:06.314718 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34243/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:36809
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33927
--webserver_port=39245
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:34243/kms with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:53:06.323228 12679 log.cc:824] T 00000000000000000000000000000000 P 8f2e42251ae2400fa105f2dfc79d506c: Log is configured to *not* fsync() on all Append() calls
I20260926 20:53:06.325150 12679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f2e42251ae2400fa105f2dfc79d506c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:53:06.325462 12679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f2e42251ae2400fa105f2dfc79d506c: Bootstrap complete.
I20260926 20:53:06.328341 12679 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f2e42251ae2400fa105f2dfc79d506c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f2e42251ae2400fa105f2dfc79d506c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33927 } }
I20260926 20:53:06.328616 12679 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f2e42251ae2400fa105f2dfc79d506c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2e42251ae2400fa105f2dfc79d506c, State: Initialized, Role: FOLLOWER
I20260926 20:53:06.328835 12679 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f2e42251ae2400fa105f2dfc79d506c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f2e42251ae2400fa105f2dfc79d506c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33927 } }
I20260926 20:53:06.328933 12679 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f2e42251ae2400fa105f2dfc79d506c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:53:06.328998 12679 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f2e42251ae2400fa105f2dfc79d506c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:53:06.329056 12679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f2e42251ae2400fa105f2dfc79d506c [term 1 FOLLOWER]: Advancing to term 2
I20260926 20:53:06.330152 12679 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f2e42251ae2400fa105f2dfc79d506c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f2e42251ae2400fa105f2dfc79d506c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33927 } }
I20260926 20:53:06.330281 12679 leader_election.cc:302] T 00000000000000000000000000000000 P 8f2e42251ae2400fa105f2dfc79d506c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f2e42251ae2400fa105f2dfc79d506c; no voters:
I20260926 20:53:06.330488 12679 leader_election.cc:288] T 00000000000000000000000000000000 P 8f2e42251ae2400fa105f2dfc79d506c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 20:53:06.330557 12683 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f2e42251ae2400fa105f2dfc79d506c [term 2 FOLLOWER]: Leader election won for term 2
I20260926 20:53:06.330699 12683 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f2e42251ae2400fa105f2dfc79d506c [term 2 LEADER]: Becoming Leader. State: Replica: 8f2e42251ae2400fa105f2dfc79d506c, State: Running, Role: LEADER
I20260926 20:53:06.330788 12679 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f2e42251ae2400fa105f2dfc79d506c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:53:06.330802 12683 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f2e42251ae2400fa105f2dfc79d506c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f2e42251ae2400fa105f2dfc79d506c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33927 } }
I20260926 20:53:06.331063 12683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f2e42251ae2400fa105f2dfc79d506c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f2e42251ae2400fa105f2dfc79d506c. Latest consensus state: current_term: 2 leader_uuid: "8f2e42251ae2400fa105f2dfc79d506c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f2e42251ae2400fa105f2dfc79d506c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33927 } } }
I20260926 20:53:06.331127 12683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f2e42251ae2400fa105f2dfc79d506c [sys.catalog]: This master's current role is: LEADER
I20260926 20:53:06.331262 12684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f2e42251ae2400fa105f2dfc79d506c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8f2e42251ae2400fa105f2dfc79d506c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f2e42251ae2400fa105f2dfc79d506c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33927 } } }
I20260926 20:53:06.331331 12687 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:53:06.331403 12684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f2e42251ae2400fa105f2dfc79d506c [sys.catalog]: This master's current role is: LEADER
I20260926 20:53:06.331889 12687 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:53:06.332023 12687 catalog_manager.cc:1259] Loaded cluster ID: fbb492a8c1214558beab2c6d947bd2ca
I20260926 20:53:06.332057 12687 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:53:06.332947 12679 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 20:53:06.333263 12687 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:53:06.333467 12687 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8f2e42251ae2400fa105f2dfc79d506c: Loaded TSK: 0
I20260926 20:53:06.333685 12687 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 20:53:06.422618 12681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:53:06.422878 12681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:53:06.422922 12681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:53:06.424392 12681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:53:06.424461 12681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:53:06.424510 12681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:53:06.426257 12681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:34243/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:36809
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=39245
--tserver_master_addrs=127.0.98.62:33927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:53:06.426535 12681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:53:06.426752 12681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:53:06.432646 12719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:06.432652 12716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:06.432652 12717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:06.433394 12681 server_base.cc:1034] running on GCE node
I20260926 20:53:06.433620 12681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:53:06.433903 12681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:53:06.435060 12681 hybrid_clock.cc:648] HybridClock initialized: now 1790455986435050 us; error 31 us; skew 500 ppm
Sep 26 20:53:06 dist-test-slave-hwlj krb5kdc[11647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:53:06.437672 12681 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 20:53:06.438210 12681 webserver.cc:460] Webserver started at http://127.0.98.1:39245/ using document root <none> and password file <none>
I20260926 20:53:06.438449 12681 fs_manager.cc:359] Metadata directory not provided
I20260926 20:53:06.438529 12681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 20:53:06 dist-test-slave-hwlj krb5kdc[11647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 20:53:06.456355 12681 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:53:06.458552 12727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:06.458714 12681 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 20:53:06.458794 12681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-0/wal
uuid: "e1969346889647daa3791a84afa2f6d4"
format_stamp: "Formatted at 2026-09-26 20:53:05 on dist-test-slave-hwlj"
server_key: "3f9468ddbd192730a7b5e0fe324d0ef8"
server_key_iv: "6e08100449372b4d14277d015b4cd313"
server_key_version: "kuduclusterkey@0"
I20260926 20:53:06.459190 12681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:53:06.472685 12681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:53:06.475128 12681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:53:06.475311 12681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:53:06.475536 12681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:53:06.475966 12681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:53:06.476028 12681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:06.476068 12681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:53:06.476094 12681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:06.482560 12681 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36809
I20260926 20:53:06.483058 12681 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:53:06.484124 12840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36809 every 8 connection(s)
I20260926 20:53:06.491035 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 12681
I20260926 20:53:06.491303 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34243/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42389
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33927
--webserver_port=41603
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:34243/kms with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:53:06 dist-test-slave-hwlj krb5kdc[11647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:53:06.502898 12843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:06.485222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33927 (local address 127.0.98.1:60549)
0926 20:53:06.485692 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:06.485728 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:06.486233 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:06.486676 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:06.486680 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:06.486898 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:06.487080 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:06.487084 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:06.488257 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:06.488258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:06.489044 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:06.489046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:06.489646 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:06.489647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:06.489766 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:06.492281 (+ 2515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:06.492293 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:06.497856 (+ 5563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:53:06.500701 (+ 2845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:06.500708 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:06.500711 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:06.501083 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:06.501401 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:06.501402 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:06.501402 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:06.501555 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:06.502103 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:53:06.502110 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:53:06.502306 (+ 196us) client_negotiation.cc:769] Sending connection context
0926 20:53:06.502446 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 20:53:06.502634 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":79,"threads_started":1}
I20260926 20:53:06.503578 12844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:06.485272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60549 (local address 127.0.98.62:33927)
0926 20:53:06.485477 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:06.485487 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:06.485844 (+ 357us) server_negotiation.cc:407] Connection header received
0926 20:53:06.486465 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:06.486470 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:06.486512 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:06.486639 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:06.487187 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:06.488159 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:06.489142 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:06.489537 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:06.489659 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:06.498022 (+ 8363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:06.498028 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:06.498029 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:06.498044 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:06.500538 (+ 2494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:06.501190 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:06.501194 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:06.501195 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:06.501318 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:06.501635 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:06.501636 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:06.501636 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:06.502007 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:06.502107 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:06.502528 (+ 421us) server_negotiation.cc:299] Negotiation successful
0926 20:53:06.502691 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":72,"threads_started":1}
I20260926 20:53:06.504321 12841 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33927
I20260926 20:53:06.504434 12841 heartbeater.cc:464] Registering TS with master...
I20260926 20:53:06.504659 12841 heartbeater.cc:511] Master 127.0.98.62:33927 requested a full tablet report, sending...
W20260926 20:53:06.505597 12841 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33927 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 20:53:06.505681 12841 heartbeater.cc:464] Registering TS with master...
I20260926 20:53:06.505761 12841 heartbeater.cc:511] Master 127.0.98.62:33927 requested a full tablet report, sending...
I20260926 20:53:06.506112 12841 heartbeater.cc:464] Registering TS with master...
I20260926 20:53:06.506201 12841 heartbeater.cc:511] Master 127.0.98.62:33927 requested a full tablet report, sending...
W20260926 20:53:06.506510 12841 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 20:53:06.606364 12845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:53:06.606666 12845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:53:06.606725 12845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:53:06.608817 12845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:53:06.608886 12845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:53:06.608932 12845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:53:06.610725 12845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:34243/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42389
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=41603
--tserver_master_addrs=127.0.98.62:33927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:53:06.611006 12845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:53:06.611246 12845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:53:06.617213 12850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:06.617360 12853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:06.617563 12851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:06.623222 12845 server_base.cc:1034] running on GCE node
I20260926 20:53:06.623498 12845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:53:06.623900 12845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:53:06.625069 12845 hybrid_clock.cc:648] HybridClock initialized: now 1790455986625047 us; error 39 us; skew 500 ppm
Sep 26 20:53:06 dist-test-slave-hwlj krb5kdc[11647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:53:06.628284 12845 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 20:53:06.628902 12845 webserver.cc:460] Webserver started at http://127.0.98.2:41603/ using document root <none> and password file <none>
I20260926 20:53:06.629197 12845 fs_manager.cc:359] Metadata directory not provided
I20260926 20:53:06.629285 12845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 20:53:06 dist-test-slave-hwlj krb5kdc[11647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 20:53:06.663293 12845 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260926 20:53:06.664193 12861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:06.664395 12845 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 20:53:06.664476 12845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-1/wal
uuid: "1d3a0e73ec5945fdb930973744c5f039"
format_stamp: "Formatted at 2026-09-26 20:53:05 on dist-test-slave-hwlj"
server_key: "7bb54c127b364a4c24d0ba0faa024978"
server_key_iv: "ec8a64fb3edf06a8a31af457ed39e6ae"
server_key_version: "kuduclusterkey@0"
I20260926 20:53:06.664913 12845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:53:06.711498 12845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:53:06.713814 12845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:53:06.713970 12845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:53:06.714210 12845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:53:06.714596 12845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:53:06.714633 12845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:06.714656 12845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:53:06.714669 12845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:06.721557 12845 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42389
I20260926 20:53:06.721647 12974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42389 every 8 connection(s)
I20260926 20:53:06.721980 12845 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
I20260926 20:53:06.723510 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 12845
I20260926 20:53:06.723827 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34243/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37203
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33927
--webserver_port=37675
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:34243/kms with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:53:06 dist-test-slave-hwlj krb5kdc[11647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:53:06.734118 12977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:06.722789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33927 (local address 127.0.98.2:37133)
0926 20:53:06.723072 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:06.723119 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:06.724145 (+ 1026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:06.724460 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:06.724465 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:06.724728 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:06.725074 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:06.725082 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:06.726128 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:06.726129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:06.727465 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:06.727469 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:06.727968 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:06.727970 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:06.728124 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:06.728796 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:06.728810 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:06.730545 (+ 1735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:53:06.732345 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:06.732353 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:06.732365 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:06.732693 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:06.732929 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:06.732930 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:06.732931 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:06.733077 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:06.733393 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:53:06.733400 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:53:06.733589 (+ 189us) client_negotiation.cc:769] Sending connection context
0926 20:53:06.733701 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 20:53:06.733868 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":86,"threads_started":1}
I20260926 20:53:06.734654 12844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:06.722843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37133 (local address 127.0.98.62:33927)
0926 20:53:06.722902 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:06.722904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:06.723224 (+ 320us) server_negotiation.cc:407] Connection header received
0926 20:53:06.724275 (+ 1051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:06.724277 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:06.724295 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:06.724366 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:06.725204 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:06.725990 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:06.727570 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:06.727838 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:06.727943 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:06.730663 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:06.730665 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:06.730665 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:06.730673 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:06.732242 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:06.732793 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:06.732795 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:06.732795 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:06.732860 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:06.733151 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:06.733152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:06.733152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:06.733292 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:06.733353 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:06.734470 (+ 1117us) server_negotiation.cc:299] Negotiation successful
0926 20:53:06.734569 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 20:53:06.735441 12975 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33927
I20260926 20:53:06.735567 12975 heartbeater.cc:464] Registering TS with master...
I20260926 20:53:06.735809 12975 heartbeater.cc:511] Master 127.0.98.62:33927 requested a full tablet report, sending...
W20260926 20:53:06.736461 12975 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33927 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 20:53:06.736544 12975 heartbeater.cc:464] Registering TS with master...
I20260926 20:53:06.736620 12975 heartbeater.cc:511] Master 127.0.98.62:33927 requested a full tablet report, sending...
I20260926 20:53:06.736903 12975 heartbeater.cc:464] Registering TS with master...
I20260926 20:53:06.736975 12975 heartbeater.cc:511] Master 127.0.98.62:33927 requested a full tablet report, sending...
W20260926 20:53:06.737265 12975 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 20:53:06.854333 12978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:53:06.854676 12978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:53:06.854719 12978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:53:06.856616 12978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:53:06.856695 12978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:53:06.856737 12978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:53:06.859161 12978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:34243/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37203
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=37675
--tserver_master_addrs=127.0.98.62:33927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:53:06.859469 12978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:53:06.859810 12978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:53:06.869211 12985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:06.869253 12984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:06.870141 12987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:06.870446 12978 server_base.cc:1034] running on GCE node
I20260926 20:53:06.870695 12978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:53:06.871013 12978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:53:06.872200 12978 hybrid_clock.cc:648] HybridClock initialized: now 1790455986872182 us; error 29 us; skew 500 ppm
Sep 26 20:53:06 dist-test-slave-hwlj krb5kdc[11647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:53:06.877040 12978 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 20:53:06.877673 12978 webserver.cc:460] Webserver started at http://127.0.98.3:37675/ using document root <none> and password file <none>
I20260926 20:53:06.877969 12978 fs_manager.cc:359] Metadata directory not provided
I20260926 20:53:06.878059 12978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 20:53:06 dist-test-slave-hwlj krb5kdc[11647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 20:53:06.891492 12978 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:53:06.892295 12995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:06.892445 12978 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:06.892534 12978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-2/wal
uuid: "b215d4f3878b4a818486c492098002f3"
format_stamp: "Formatted at 2026-09-26 20:53:06 on dist-test-slave-hwlj"
server_key: "c842473da7caa967a61ad39c369d6210"
server_key_iv: "97d43eae909f21f7216c54d0a43cbdaf"
server_key_version: "kuduclusterkey@0"
I20260926 20:53:06.892900 12978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:53:06.914060 12978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:53:06.916332 12978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:53:06.916497 12978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:53:06.916779 12978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:53:06.917163 12978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:53:06.917215 12978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:06.917251 12978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:53:06.917275 12978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:06.924264 12978 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37203
I20260926 20:53:06.924743 12978 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
I20260926 20:53:06.925266 13108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37203 every 8 connection(s)
I20260926 20:53:06.926258 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 12978
Sep 26 20:53:06 dist-test-slave-hwlj krb5kdc[11647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:53:06.936616 12844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:06.925866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45979 (local address 127.0.98.62:33927)
0926 20:53:06.925933 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:06.925935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:06.926352 (+ 417us) server_negotiation.cc:407] Connection header received
0926 20:53:06.927095 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:06.927096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:06.927109 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:06.927187 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:06.927968 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:06.928771 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:06.930278 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:06.930584 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:06.930644 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:06.933338 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:06.933340 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:06.933340 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:06.933348 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:06.934884 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:06.935428 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:06.935429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:06.935430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:06.935504 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:06.935896 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:06.935897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:06.935897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:06.936045 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:06.936137 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:06.936442 (+ 305us) server_negotiation.cc:299] Negotiation successful
0926 20:53:06.936526 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 20:53:06.936829 13111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:06.925856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33927 (local address 127.0.98.3:45979)
0926 20:53:06.926202 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:06.926253 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:06.926990 (+ 737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:06.927292 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:06.927296 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:06.927545 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:06.927855 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:06.927862 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:06.928880 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:06.928882 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:06.930166 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:06.930170 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:06.930720 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:06.930721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:06.930863 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:06.931477 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:06.931488 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:06.933214 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:53:06.934992 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:06.935001 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:06.935003 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:06.935347 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:06.935582 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:06.935583 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:06.935583 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:06.935820 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:06.936126 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:53:06.936133 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:53:06.936297 (+ 164us) client_negotiation.cc:769] Sending connection context
0926 20:53:06.936405 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 20:53:06.936558 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":81,"threads_started":1}
I20260926 20:53:06.937573 13109 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33927
I20260926 20:53:06.937690 13109 heartbeater.cc:464] Registering TS with master...
I20260926 20:53:06.937904 13109 heartbeater.cc:511] Master 127.0.98.62:33927 requested a full tablet report, sending...
W20260926 20:53:06.938550 13109 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33927 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 20:53:06.938630 13109 heartbeater.cc:464] Registering TS with master...
I20260926 20:53:06.938921 13109 heartbeater.cc:511] Master 127.0.98.62:33927 requested a full tablet report, sending...
I20260926 20:53:06.939277 13109 heartbeater.cc:464] Registering TS with master...
I20260926 20:53:06.939368 13109 heartbeater.cc:511] Master 127.0.98.62:33927 requested a full tablet report, sending...
W20260926 20:53:06.939723 13109 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 20:53:06.941366 13112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:06.926649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37246 (local address 127.0.98.62:33927)
0926 20:53:06.927136 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:06.927137 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:06.927143 (+ 6us) server_negotiation.cc:407] Connection header received
0926 20:53:06.927182 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:06.927183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:06.927195 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:06.927255 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:06.934279 (+ 7024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:06.935026 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:06.936164 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:06.936495 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:06.936566 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:06.938445 (+ 1879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:06.938447 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:06.938447 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:06.938455 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:06.940105 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:06.940497 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:06.940498 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:06.940498 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:06.940550 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:06.940777 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:06.940778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:06.940778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:06.940903 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:06.941052 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:06.941168 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 20:53:06.941246 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":49,"threads_started":1}
Sep 26 20:53:07 dist-test-slave-hwlj krb5kdc[11647](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790455987, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:53:07.506721 12841 heartbeater.cc:464] Registering TS with master...
I20260926 20:53:07.506879 12841 heartbeater.cc:511] Master 127.0.98.62:33927 requested a full tablet report, sending...
I20260926 20:53:07.507282 12643 ts_manager.cc:194] Registered new tserver with Master: e1969346889647daa3791a84afa2f6d4 (127.0.98.1:36809)
I20260926 20:53:07.508323 12643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60549
I20260926 20:53:07.737527 12975 heartbeater.cc:464] Registering TS with master...
I20260926 20:53:07.737702 12975 heartbeater.cc:511] Master 127.0.98.62:33927 requested a full tablet report, sending...
I20260926 20:53:07.738137 12643 ts_manager.cc:194] Registered new tserver with Master: 1d3a0e73ec5945fdb930973744c5f039 (127.0.98.2:42389)
I20260926 20:53:07.738682 12643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37133
Sep 26 20:53:07 dist-test-slave-hwlj krb5kdc[11647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455987, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 20:53:07.939934 13109 heartbeater.cc:464] Registering TS with master...
I20260926 20:53:07.940104 13109 heartbeater.cc:511] Master 127.0.98.62:33927 requested a full tablet report, sending...
I20260926 20:53:07.940878 12643 ts_manager.cc:194] Registered new tserver with Master: b215d4f3878b4a818486c492098002f3 (127.0.98.3:37203)
I20260926 20:53:07.941429 12643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45979
I20260926 20:53:07.944404 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:53:07.953754 13130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:07.945868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37254 (local address 127.0.98.62:33927)
0926 20:53:07.946130 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:07.946132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:07.946317 (+ 185us) server_negotiation.cc:407] Connection header received
0926 20:53:07.946341 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:07.946342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:07.946354 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:07.946410 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:07.947145 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:07.947947 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:07.948924 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:07.949214 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:07.949265 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:07.950720 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:07.950722 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:07.950722 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:07.950729 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:07.952389 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:07.952739 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:07.952740 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:07.952740 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:07.952792 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:07.952982 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:07.952982 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:07.952983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:07.953100 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:07.953189 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:07.953581 (+ 392us) server_negotiation.cc:299] Negotiation successful
0926 20:53:07.953660 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":63,"threads_started":1}
I20260926 20:53:07.955971 12643 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37254:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:53:07.958159 12643 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:53:07.973748 13136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:07.965508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37203 (local address 127.0.0.1:49194)
0926 20:53:07.967953 (+ 2445us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:07.968034 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:07.968387 (+ 353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:07.968746 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:07.968749 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:07.968751 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:53:07.968834 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:07.968836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:07.970217 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:07.970218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:07.971910 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:07.971913 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:07.973428 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:07.973428 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:07.973511 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:07.973533 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 20:53:07.973602 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 20:53:07.973658 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2414,"spinlock_wait_cycles":896,"thread_start_us":87,"threads_started":1}
I20260926 20:53:07.977476 13139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:07.965863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49194 (local address 127.0.98.3:37203)
0926 20:53:07.968532 (+ 2669us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:07.968537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:07.968547 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:53:07.968581 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:07.968583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:07.968599 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:07.968665 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:53:07.968959 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:07.970105 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:07.972163 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:07.972953 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:07.973030 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:07.973115 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:07.977330 (+ 4215us) server_negotiation.cc:299] Negotiation successful
0926 20:53:07.977387 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2634,"thread_start_us":50,"threads_started":1}
I20260926 20:53:07.979012 13136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:07.966079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36809 (local address 127.0.0.1:46940)
0926 20:53:07.974191 (+ 8112us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:07.974201 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:07.974279 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:07.974540 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:07.974541 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:07.974542 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:53:07.974574 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:07.974575 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:07.975804 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:07.975805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:07.977217 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:07.977221 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:07.978783 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:07.978784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:07.978846 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:07.978849 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:53:07.978895 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 20:53:07.978935 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8100,"thread_start_us":68,"threads_started":1}
I20260926 20:53:07.979408 13141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:07.966188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46940 (local address 127.0.98.1:36809)
0926 20:53:07.967875 (+ 1687us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:07.967882 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:07.974251 (+ 6369us) server_negotiation.cc:407] Connection header received
0926 20:53:07.974362 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:07.974365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:07.974382 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:07.974437 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:53:07.974641 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:07.975684 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:07.977788 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:07.978579 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:07.978633 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:07.978716 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:07.979285 (+ 569us) server_negotiation.cc:299] Negotiation successful
0926 20:53:07.979331 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1649,"thread_start_us":62,"threads_started":1}
I20260926 20:53:07.980401 13043 tablet_service.cc:1467] Processing CreateTablet for tablet cc5b618a35d742199d607d1b47153367 (DEFAULT_TABLE table=test-table [id=3145d36e333a4659a3a5c9b4e0091691]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:53:07.980808 13043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc5b618a35d742199d607d1b47153367. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:53:07.985988 12775 tablet_service.cc:1467] Processing CreateTablet for tablet cc5b618a35d742199d607d1b47153367 (DEFAULT_TABLE table=test-table [id=3145d36e333a4659a3a5c9b4e0091691]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:53:07.986356 12775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc5b618a35d742199d607d1b47153367. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:53:07.989421 13142 tablet_bootstrap.cc:492] T cc5b618a35d742199d607d1b47153367 P b215d4f3878b4a818486c492098002f3: Bootstrap starting.
I20260926 20:53:07.990522 13142 tablet_bootstrap.cc:654] T cc5b618a35d742199d607d1b47153367 P b215d4f3878b4a818486c492098002f3: Neither blocks nor log segments found. Creating new log.
I20260926 20:53:07.990942 13137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:07.965508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42389 (local address 127.0.0.1:40200)
0926 20:53:07.972030 (+ 6522us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:07.972042 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:07.972080 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:07.985780 (+ 13700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:07.985781 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:07.985783 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:53:07.985838 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:07.985840 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:07.987122 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:07.987123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:07.988274 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:07.988277 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:07.990691 (+ 2414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:07.990693 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:07.990773 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:07.990778 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:53:07.990811 (+ 33us) client_negotiation.cc:240] Negotiation successful
0926 20:53:07.990859 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6504,"mutex_wait_us":2,"thread_start_us":77,"threads_started":1}
I20260926 20:53:07.991546 13138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:07.965845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40200 (local address 127.0.98.2:42389)
0926 20:53:07.985597 (+ 19752us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:07.985602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:07.985614 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:53:07.985652 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:07.985654 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:07.985671 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:07.985906 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:53:07.985985 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:07.987018 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:07.988392 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:07.989773 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:07.991310 (+ 1537us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:07.991401 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:07.991428 (+ 27us) server_negotiation.cc:299] Negotiation successful
0926 20:53:07.991470 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19711,"thread_start_us":65,"threads_started":1}
I20260926 20:53:07.992370 12900 tablet_service.cc:1467] Processing CreateTablet for tablet cc5b618a35d742199d607d1b47153367 (DEFAULT_TABLE table=test-table [id=3145d36e333a4659a3a5c9b4e0091691]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:53:07.992704 13142 log.cc:824] T cc5b618a35d742199d607d1b47153367 P b215d4f3878b4a818486c492098002f3: Log is configured to *not* fsync() on all Append() calls
I20260926 20:53:07.992748 12900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc5b618a35d742199d607d1b47153367. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:53:07.993511 13142 tablet_bootstrap.cc:492] T cc5b618a35d742199d607d1b47153367 P b215d4f3878b4a818486c492098002f3: No bootstrap required, opened a new log
I20260926 20:53:07.993571 13142 ts_tablet_manager.cc:1397] T cc5b618a35d742199d607d1b47153367 P b215d4f3878b4a818486c492098002f3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 20:53:07.996206 13142 raft_consensus.cc:348] T cc5b618a35d742199d607d1b47153367 P b215d4f3878b4a818486c492098002f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215d4f3878b4a818486c492098002f3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37203 } } peers { permanent_uuid: "e1969346889647daa3791a84afa2f6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36809 } } peers { permanent_uuid: "1d3a0e73ec5945fdb930973744c5f039" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } }
I20260926 20:53:07.997025 13142 raft_consensus.cc:374] T cc5b618a35d742199d607d1b47153367 P b215d4f3878b4a818486c492098002f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:53:07.997129 13142 raft_consensus.cc:729] T cc5b618a35d742199d607d1b47153367 P b215d4f3878b4a818486c492098002f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b215d4f3878b4a818486c492098002f3, State: Initialized, Role: FOLLOWER
I20260926 20:53:07.997356 13142 consensus_queue.cc:260] T cc5b618a35d742199d607d1b47153367 P b215d4f3878b4a818486c492098002f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215d4f3878b4a818486c492098002f3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37203 } } peers { permanent_uuid: "e1969346889647daa3791a84afa2f6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36809 } } peers { permanent_uuid: "1d3a0e73ec5945fdb930973744c5f039" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } }
I20260926 20:53:07.997731 13142 ts_tablet_manager.cc:1428] T cc5b618a35d742199d607d1b47153367 P b215d4f3878b4a818486c492098002f3: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260926 20:53:07.997906 13109 heartbeater.cc:503] Master 127.0.98.62:33927 was elected leader, sending a full tablet report...
I20260926 20:53:08.003984 13145 tablet_bootstrap.cc:492] T cc5b618a35d742199d607d1b47153367 P 1d3a0e73ec5945fdb930973744c5f039: Bootstrap starting.
I20260926 20:53:08.004280 13143 tablet_bootstrap.cc:492] T cc5b618a35d742199d607d1b47153367 P e1969346889647daa3791a84afa2f6d4: Bootstrap starting.
I20260926 20:53:08.005594 13145 tablet_bootstrap.cc:654] T cc5b618a35d742199d607d1b47153367 P 1d3a0e73ec5945fdb930973744c5f039: Neither blocks nor log segments found. Creating new log.
I20260926 20:53:08.006258 13143 tablet_bootstrap.cc:654] T cc5b618a35d742199d607d1b47153367 P e1969346889647daa3791a84afa2f6d4: Neither blocks nor log segments found. Creating new log.
I20260926 20:53:08.006632 13143 log.cc:824] T cc5b618a35d742199d607d1b47153367 P e1969346889647daa3791a84afa2f6d4: Log is configured to *not* fsync() on all Append() calls
I20260926 20:53:08.006996 13145 log.cc:824] T cc5b618a35d742199d607d1b47153367 P 1d3a0e73ec5945fdb930973744c5f039: Log is configured to *not* fsync() on all Append() calls
I20260926 20:53:08.008306 13143 tablet_bootstrap.cc:492] T cc5b618a35d742199d607d1b47153367 P e1969346889647daa3791a84afa2f6d4: No bootstrap required, opened a new log
I20260926 20:53:08.008431 13143 ts_tablet_manager.cc:1397] T cc5b618a35d742199d607d1b47153367 P e1969346889647daa3791a84afa2f6d4: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 20:53:08.011055 13143 raft_consensus.cc:348] T cc5b618a35d742199d607d1b47153367 P e1969346889647daa3791a84afa2f6d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215d4f3878b4a818486c492098002f3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37203 } } peers { permanent_uuid: "e1969346889647daa3791a84afa2f6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36809 } } peers { permanent_uuid: "1d3a0e73ec5945fdb930973744c5f039" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } }
I20260926 20:53:08.011305 13143 raft_consensus.cc:374] T cc5b618a35d742199d607d1b47153367 P e1969346889647daa3791a84afa2f6d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:53:08.011384 13143 raft_consensus.cc:729] T cc5b618a35d742199d607d1b47153367 P e1969346889647daa3791a84afa2f6d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1969346889647daa3791a84afa2f6d4, State: Initialized, Role: FOLLOWER
I20260926 20:53:08.011579 13143 consensus_queue.cc:260] T cc5b618a35d742199d607d1b47153367 P e1969346889647daa3791a84afa2f6d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215d4f3878b4a818486c492098002f3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37203 } } peers { permanent_uuid: "e1969346889647daa3791a84afa2f6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36809 } } peers { permanent_uuid: "1d3a0e73ec5945fdb930973744c5f039" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } }
I20260926 20:53:08.011951 13143 ts_tablet_manager.cc:1428] T cc5b618a35d742199d607d1b47153367 P e1969346889647daa3791a84afa2f6d4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:53:08.012156 12841 heartbeater.cc:503] Master 127.0.98.62:33927 was elected leader, sending a full tablet report...
I20260926 20:53:08.012528 13145 tablet_bootstrap.cc:492] T cc5b618a35d742199d607d1b47153367 P 1d3a0e73ec5945fdb930973744c5f039: No bootstrap required, opened a new log
I20260926 20:53:08.012593 13145 ts_tablet_manager.cc:1397] T cc5b618a35d742199d607d1b47153367 P 1d3a0e73ec5945fdb930973744c5f039: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260926 20:53:08.016255 13145 raft_consensus.cc:348] T cc5b618a35d742199d607d1b47153367 P 1d3a0e73ec5945fdb930973744c5f039 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215d4f3878b4a818486c492098002f3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37203 } } peers { permanent_uuid: "e1969346889647daa3791a84afa2f6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36809 } } peers { permanent_uuid: "1d3a0e73ec5945fdb930973744c5f039" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } }
I20260926 20:53:08.016506 13145 raft_consensus.cc:374] T cc5b618a35d742199d607d1b47153367 P 1d3a0e73ec5945fdb930973744c5f039 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:53:08.016590 13145 raft_consensus.cc:729] T cc5b618a35d742199d607d1b47153367 P 1d3a0e73ec5945fdb930973744c5f039 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d3a0e73ec5945fdb930973744c5f039, State: Initialized, Role: FOLLOWER
I20260926 20:53:08.016898 13145 consensus_queue.cc:260] T cc5b618a35d742199d607d1b47153367 P 1d3a0e73ec5945fdb930973744c5f039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215d4f3878b4a818486c492098002f3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37203 } } peers { permanent_uuid: "e1969346889647daa3791a84afa2f6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36809 } } peers { permanent_uuid: "1d3a0e73ec5945fdb930973744c5f039" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } }
I20260926 20:53:08.017261 13145 ts_tablet_manager.cc:1428] T cc5b618a35d742199d607d1b47153367 P 1d3a0e73ec5945fdb930973744c5f039: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260926 20:53:08.017410 12975 heartbeater.cc:503] Master 127.0.98.62:33927 was elected leader, sending a full tablet report...
I20260926 20:53:08.123391 13150 raft_consensus.cc:482] T cc5b618a35d742199d607d1b47153367 P 1d3a0e73ec5945fdb930973744c5f039 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:53:08.123554 13150 raft_consensus.cc:504] T cc5b618a35d742199d607d1b47153367 P 1d3a0e73ec5945fdb930973744c5f039 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215d4f3878b4a818486c492098002f3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37203 } } peers { permanent_uuid: "e1969346889647daa3791a84afa2f6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36809 } } peers { permanent_uuid: "1d3a0e73ec5945fdb930973744c5f039" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } }
I20260926 20:53:08.124019 13150 leader_election.cc:288] T cc5b618a35d742199d607d1b47153367 P 1d3a0e73ec5945fdb930973744c5f039 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b215d4f3878b4a818486c492098002f3 (127.0.98.3:37203), e1969346889647daa3791a84afa2f6d4 (127.0.98.1:36809)
I20260926 20:53:08.128571 13139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:08.124696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40839 (local address 127.0.98.3:37203)
0926 20:53:08.124799 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:08.124802 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:08.124812 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:53:08.124841 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:08.124842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:08.124852 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:08.124919 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:53:08.125127 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:08.125852 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:08.127544 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:08.128184 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:08.128268 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:08.128281 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:08.128455 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 20:53:08.128494 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 20:53:08.128605 13151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:08.124229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37203 (local address 127.0.98.2:40839)
0926 20:53:08.124401 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:08.124416 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:08.124492 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:08.124946 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:08.124947 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:08.124950 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:53:08.125052 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:08.125054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:08.125940 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:08.125942 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:08.127255 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:08.127259 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:08.128289 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:08.128291 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:08.128373 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:08.128379 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 20:53:08.128447 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 20:53:08.128493 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":100,"threads_started":1}
I20260926 20:53:08.129428 13053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5b618a35d742199d607d1b47153367" candidate_uuid: "1d3a0e73ec5945fdb930973744c5f039" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b215d4f3878b4a818486c492098002f3" is_pre_election: true
I20260926 20:53:08.129597 13053 raft_consensus.cc:2454] T cc5b618a35d742199d607d1b47153367 P b215d4f3878b4a818486c492098002f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d3a0e73ec5945fdb930973744c5f039 in term 0.
I20260926 20:53:08.129824 12863 leader_election.cc:302] T cc5b618a35d742199d607d1b47153367 P 1d3a0e73ec5945fdb930973744c5f039 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d3a0e73ec5945fdb930973744c5f039, b215d4f3878b4a818486c492098002f3; no voters:
I20260926 20:53:08.129952 13150 raft_consensus.cc:2781] T cc5b618a35d742199d607d1b47153367 P 1d3a0e73ec5945fdb930973744c5f039 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:53:08.129989 13150 raft_consensus.cc:482] T cc5b618a35d742199d607d1b47153367 P 1d3a0e73ec5945fdb930973744c5f039 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:53:08.130010 13150 raft_consensus.cc:3037] T cc5b618a35d742199d607d1b47153367 P 1d3a0e73ec5945fdb930973744c5f039 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:53:08.130908 13150 raft_consensus.cc:504] T cc5b618a35d742199d607d1b47153367 P 1d3a0e73ec5945fdb930973744c5f039 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215d4f3878b4a818486c492098002f3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37203 } } peers { permanent_uuid: "e1969346889647daa3791a84afa2f6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36809 } } peers { permanent_uuid: "1d3a0e73ec5945fdb930973744c5f039" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } }
I20260926 20:53:08.131040 13150 leader_election.cc:288] T cc5b618a35d742199d607d1b47153367 P 1d3a0e73ec5945fdb930973744c5f039 [CANDIDATE]: Term 1 election: Requested vote from peers b215d4f3878b4a818486c492098002f3 (127.0.98.3:37203), e1969346889647daa3791a84afa2f6d4 (127.0.98.1:36809)
I20260926 20:53:08.132118 13053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5b618a35d742199d607d1b47153367" candidate_uuid: "1d3a0e73ec5945fdb930973744c5f039" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b215d4f3878b4a818486c492098002f3"
I20260926 20:53:08.132215 13053 raft_consensus.cc:3037] T cc5b618a35d742199d607d1b47153367 P b215d4f3878b4a818486c492098002f3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:53:08.133092 13053 raft_consensus.cc:2454] T cc5b618a35d742199d607d1b47153367 P b215d4f3878b4a818486c492098002f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d3a0e73ec5945fdb930973744c5f039 in term 1.
I20260926 20:53:08.133363 12863 leader_election.cc:302] T cc5b618a35d742199d607d1b47153367 P 1d3a0e73ec5945fdb930973744c5f039 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d3a0e73ec5945fdb930973744c5f039, b215d4f3878b4a818486c492098002f3; no voters:
I20260926 20:53:08.133772 13150 raft_consensus.cc:2781] T cc5b618a35d742199d607d1b47153367 P 1d3a0e73ec5945fdb930973744c5f039 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:53:08.133939 13150 raft_consensus.cc:686] T cc5b618a35d742199d607d1b47153367 P 1d3a0e73ec5945fdb930973744c5f039 [term 1 LEADER]: Becoming Leader. State: Replica: 1d3a0e73ec5945fdb930973744c5f039, State: Running, Role: LEADER
I20260926 20:53:08.134032 13150 consensus_queue.cc:237] T cc5b618a35d742199d607d1b47153367 P 1d3a0e73ec5945fdb930973744c5f039 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215d4f3878b4a818486c492098002f3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37203 } } peers { permanent_uuid: "e1969346889647daa3791a84afa2f6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36809 } } peers { permanent_uuid: "1d3a0e73ec5945fdb930973744c5f039" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } }
I20260926 20:53:08.134891 12643 catalog_manager.cc:5495] T cc5b618a35d742199d607d1b47153367 P 1d3a0e73ec5945fdb930973744c5f039 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d3a0e73ec5945fdb930973744c5f039 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "1d3a0e73ec5945fdb930973744c5f039" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215d4f3878b4a818486c492098002f3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1969346889647daa3791a84afa2f6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d3a0e73ec5945fdb930973744c5f039" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42389 } health_report { overall_health: HEALTHY } } }
I20260926 20:53:08.136381 13151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:08.124465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36809 (local address 127.0.98.2:50101)
0926 20:53:08.128693 (+ 4228us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:08.128705 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:08.128751 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:08.129212 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:08.129213 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:08.129215 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:53:08.129249 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:08.129250 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:08.133767 (+ 4517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:08.133768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:08.134913 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:08.134915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:08.136121 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:08.136122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:08.136190 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:08.136194 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:53:08.136265 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 20:53:08.136307 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4215,"thread_start_us":86,"threads_started":1}
I20260926 20:53:08.136389 13141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:08.124754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50101 (local address 127.0.98.1:36809)
0926 20:53:08.124975 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:08.124977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:08.129101 (+ 4124us) server_negotiation.cc:407] Connection header received
0926 20:53:08.129135 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:08.129136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:08.129152 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:08.129218 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:53:08.131193 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:08.133674 (+ 2481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:08.135436 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:08.136047 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:08.136126 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:08.136138 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:08.136274 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 20:53:08.136323 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199}
I20260926 20:53:08.136757 12794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5b618a35d742199d607d1b47153367" candidate_uuid: "1d3a0e73ec5945fdb930973744c5f039" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1969346889647daa3791a84afa2f6d4" is_pre_election: true
I20260926 20:53:08.136895 12794 raft_consensus.cc:2454] T cc5b618a35d742199d607d1b47153367 P e1969346889647daa3791a84afa2f6d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d3a0e73ec5945fdb930973744c5f039 in term 0.
I20260926 20:53:08.137003 12793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5b618a35d742199d607d1b47153367" candidate_uuid: "1d3a0e73ec5945fdb930973744c5f039" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1969346889647daa3791a84afa2f6d4"
I20260926 20:53:08.137058 12793 raft_consensus.cc:3037] T cc5b618a35d742199d607d1b47153367 P e1969346889647daa3791a84afa2f6d4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:53:08.137894 12793 raft_consensus.cc:2454] T cc5b618a35d742199d607d1b47153367 P e1969346889647daa3791a84afa2f6d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d3a0e73ec5945fdb930973744c5f039 in term 1.
I20260926 20:53:08.155479 13138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:08.152275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40204 (local address 127.0.98.2:42389)
0926 20:53:08.152324 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:08.152326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:08.152336 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:53:08.152379 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:08.152380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:08.152405 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:08.152484 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:53:08.152675 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:08.153417 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:08.154485 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:08.154727 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:08.154778 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:08.154947 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:53:08.155076 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:53:08.155152 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:08.155356 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 20:53:08.155378 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 20:53:08.159264 12793 raft_consensus.cc:1261] T cc5b618a35d742199d607d1b47153367 P e1969346889647daa3791a84afa2f6d4 [term 1 FOLLOWER]: Refusing update from remote peer 1d3a0e73ec5945fdb930973744c5f039: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:53:08.159289 13053 raft_consensus.cc:1261] T cc5b618a35d742199d607d1b47153367 P b215d4f3878b4a818486c492098002f3 [term 1 FOLLOWER]: Refusing update from remote peer 1d3a0e73ec5945fdb930973744c5f039: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:53:08.159628 13153 consensus_queue.cc:1035] T cc5b618a35d742199d607d1b47153367 P 1d3a0e73ec5945fdb930973744c5f039 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b215d4f3878b4a818486c492098002f3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:53:08.159763 13150 consensus_queue.cc:1035] T cc5b618a35d742199d607d1b47153367 P 1d3a0e73ec5945fdb930973744c5f039 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1969346889647daa3791a84afa2f6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:53:08.166102 13157 mvcc.cc:205] Tried to move back new op lower bound from 7333707727493091328 to 7333707727397437440. Current Snapshot: MvccSnapshot[applied={T|T < 7333707727493091328}]
I20260926 20:53:08.166380 13160 mvcc.cc:205] Tried to move back new op lower bound from 7333707727493091328 to 7333707727397437440. Current Snapshot: MvccSnapshot[applied={T|T < 7333707727493091328}]
I20260926 20:53:08.170500 12643 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:53:08.170604 12643 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:53:08.171319 12643 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f2e42251ae2400fa105f2dfc79d506c: Sending DeleteTablet for 3 replicas of tablet cc5b618a35d742199d607d1b47153367
I20260926 20:53:08.171772 13043 tablet_service.cc:1514] Processing DeleteTablet for tablet cc5b618a35d742199d607d1b47153367 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:53:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49194
I20260926 20:53:08.171696 12900 tablet_service.cc:1514] Processing DeleteTablet for tablet cc5b618a35d742199d607d1b47153367 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:53:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40200
I20260926 20:53:08.172004 13166 tablet_replica.cc:330] T cc5b618a35d742199d607d1b47153367 P b215d4f3878b4a818486c492098002f3: stopping tablet replica
I20260926 20:53:08.172102 13166 raft_consensus.cc:2229] T cc5b618a35d742199d607d1b47153367 P b215d4f3878b4a818486c492098002f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:53:08.172221 13166 raft_consensus.cc:2258] T cc5b618a35d742199d607d1b47153367 P b215d4f3878b4a818486c492098002f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:53:08.172269 12775 tablet_service.cc:1514] Processing DeleteTablet for tablet cc5b618a35d742199d607d1b47153367 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:53:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46940
I20260926 20:53:08.172480 13168 tablet_replica.cc:330] T cc5b618a35d742199d607d1b47153367 P e1969346889647daa3791a84afa2f6d4: stopping tablet replica
I20260926 20:53:08.172571 13168 raft_consensus.cc:2229] T cc5b618a35d742199d607d1b47153367 P e1969346889647daa3791a84afa2f6d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:53:08.172679 13168 raft_consensus.cc:2258] T cc5b618a35d742199d607d1b47153367 P e1969346889647daa3791a84afa2f6d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:53:08.173202 13168 ts_tablet_manager.cc:1905] T cc5b618a35d742199d607d1b47153367 P e1969346889647daa3791a84afa2f6d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:53:08.173321 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 12681
Sep 26 20:53:08 dist-test-slave-hwlj krb5kdc[11647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455987, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260926 20:53:08.182776 12629 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36809 (error 108)
I20260926 20:53:08.183069 13166 ts_tablet_manager.cc:1905] T cc5b618a35d742199d607d1b47153367 P b215d4f3878b4a818486c492098002f3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 20:53:08.183651 13137 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:53:08.183362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36809 (local address 127.0.0.1:46948)
0926 20:53:08.183419 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:08.183431 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:08.183472 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:08.183594 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36809: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 20:53:08.183926 12629 catalog_manager.cc:4584] TS e1969346889647daa3791a84afa2f6d4 (127.0.98.1:36809): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cc5b618a35d742199d607d1b47153367: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36809: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 20:53:08.184883 13167 tablet_replica.cc:330] T cc5b618a35d742199d607d1b47153367 P 1d3a0e73ec5945fdb930973744c5f039: stopping tablet replica
I20260926 20:53:08.185045 13166 ts_tablet_manager.cc:1918] T cc5b618a35d742199d607d1b47153367 P b215d4f3878b4a818486c492098002f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:53:08.185070 13167 raft_consensus.cc:2229] T cc5b618a35d742199d607d1b47153367 P 1d3a0e73ec5945fdb930973744c5f039 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:53:08.185133 13166 log.cc:1196] T cc5b618a35d742199d607d1b47153367 P b215d4f3878b4a818486c492098002f3: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-2/wal/wals/cc5b618a35d742199d607d1b47153367
I20260926 20:53:08.185338 13167 raft_consensus.cc:2258] T cc5b618a35d742199d607d1b47153367 P 1d3a0e73ec5945fdb930973744c5f039 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:53:08.185417 13166 ts_tablet_manager.cc:1939] T cc5b618a35d742199d607d1b47153367 P b215d4f3878b4a818486c492098002f3: Deleting consensus metadata
I20260926 20:53:08.185997 13167 ts_tablet_manager.cc:1905] T cc5b618a35d742199d607d1b47153367 P 1d3a0e73ec5945fdb930973744c5f039: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:53:08.187953 13167 ts_tablet_manager.cc:1918] T cc5b618a35d742199d607d1b47153367 P 1d3a0e73ec5945fdb930973744c5f039: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:53:08.188100 13167 log.cc:1196] T cc5b618a35d742199d607d1b47153367 P 1d3a0e73ec5945fdb930973744c5f039: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455725523895-392-0/minicluster-data/ts-1/wal/wals/cc5b618a35d742199d607d1b47153367
I20260926 20:53:08.188423 13167 ts_tablet_manager.cc:1939] T cc5b618a35d742199d607d1b47153367 P 1d3a0e73ec5945fdb930973744c5f039: Deleting consensus metadata
I20260926 20:53:08.188480 12628 catalog_manager.cc:4841] TS b215d4f3878b4a818486c492098002f3 (127.0.98.3:37203): tablet cc5b618a35d742199d607d1b47153367 (table test-table [id=3145d36e333a4659a3a5c9b4e0091691]) successfully deleted
I20260926 20:53:08.188895 12627 catalog_manager.cc:4841] TS 1d3a0e73ec5945fdb930973744c5f039 (127.0.98.2:42389): tablet cc5b618a35d742199d607d1b47153367 (table test-table [id=3145d36e333a4659a3a5c9b4e0091691]) successfully deleted
I20260926 20:53:08.196231 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 12845
I20260926 20:53:08.203313 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 12978
I20260926 20:53:08.209292 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 12610
I20260926 20:53:08.221176 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 20:53:08.566273 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 20:53:08.566370 392 mini_ranger.cc:66] Stopping Ranger...
I20260926 20:53:08.911211 392 mini_ranger.cc:68] Stopped Ranger
2026-09-26 20:53:08.911 UTC [11677] LOG: received smart shutdown request
2026-09-26 20:53:08.915 UTC [11677] LOG: background worker "logical replication launcher" (PID 11686) exited with exit code 1
2026-09-26 20:53:08.916 UTC [11681] LOG: shutting down
2026-09-26 20:53:09.101 UTC [11677] LOG: database system is shut down
2026-09-26T20:53:09Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49162 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:53:09 dist-test-slave-hwlj krb5kdc[13179](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:53:09 dist-test-slave-hwlj krb5kdc[13179](info): set up 2 sockets
Sep 26 20:53:09 dist-test-slave-hwlj krb5kdc[13179](info): commencing operation
krb5kdc: starting...
W20260926 20:53:10.251566 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/krb5kdc/test-user.keytab.
Sep 26 20:53:10 dist-test-slave-hwlj krb5kdc[13179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:53:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:53:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 20:53:10.374722 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46161
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46161,127.0.98.61:37057,127.0.98.60:34115
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:53:10.455972 13192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:53:10.456216 13192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:53:10.456254 13192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:53:10.457420 13192 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:53:10.457487 13192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:53:10.457509 13192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:53:10.457530 13192 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:53:10.457562 13192 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:53:10.459110 13192 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46161,127.0.98.61:37057,127.0.98.60:34115
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:53:10.459400 13192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:53:10.459630 13192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:53:10.464458 13197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:10.464542 13198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:10.464648 13192 server_base.cc:1034] running on GCE node
W20260926 20:53:10.464542 13200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:10.464983 13192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:53:10.465291 13192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:53:10.466441 13192 hybrid_clock.cc:648] HybridClock initialized: now 1790455990466421 us; error 33 us; skew 500 ppm
Sep 26 20:53:10 dist-test-slave-hwlj krb5kdc[13179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:53:10.472361 13192 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:53:10.472790 13192 webserver.cc:460] Webserver started at http://127.0.98.62:41885/ using document root <none> and password file <none>
I20260926 20:53:10.472980 13192 fs_manager.cc:359] Metadata directory not provided
I20260926 20:53:10.473021 13192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:53:10.473127 13192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:53:10.474134 13192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/minicluster-data/master-0/data/instance:
uuid: "3b7a6988af6740fca5e8e04928c67ee1"
format_stamp: "Formatted at 2026-09-26 20:53:10 on dist-test-slave-hwlj"
server_key: "3d6241d1af7e924a3c612a87f6e1b7fb"
server_key_iv: "22df64fe3ff06a62af3d5687532a60b7"
server_key_version: "encryptionkey@0"
I20260926 20:53:10.474442 13192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/minicluster-data/master-0/wal/instance:
uuid: "3b7a6988af6740fca5e8e04928c67ee1"
format_stamp: "Formatted at 2026-09-26 20:53:10 on dist-test-slave-hwlj"
server_key: "3d6241d1af7e924a3c612a87f6e1b7fb"
server_key_iv: "22df64fe3ff06a62af3d5687532a60b7"
server_key_version: "encryptionkey@0"
I20260926 20:53:10.475697 13192 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:53:10.476495 13207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:10.476666 13192 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:53:10.476785 13192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "3b7a6988af6740fca5e8e04928c67ee1"
format_stamp: "Formatted at 2026-09-26 20:53:10 on dist-test-slave-hwlj"
server_key: "3d6241d1af7e924a3c612a87f6e1b7fb"
server_key_iv: "22df64fe3ff06a62af3d5687532a60b7"
server_key_version: "encryptionkey@0"
I20260926 20:53:10.476861 13192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:53:10.486718 13192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:53:10.488862 13192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:53:10.489023 13192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:53:10.492934 13192 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46161
I20260926 20:53:10.492964 13259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46161 every 8 connection(s)
I20260926 20:53:10.493302 13192 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:53:10.494146 13260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:53:10.495658 13260 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46161 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:53:10.499571 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 13192
I20260926 20:53:10.499671 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/minicluster-data/master-0/wal/instance
I20260926 20:53:10.499872 392 external_mini_cluster.cc:1428] Setting key 17486bfb8554b860164b00addccb9dd1
Sep 26 20:53:10 dist-test-slave-hwlj krb5kdc[13179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:53:10.506539 13263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:10.496565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39798 (local address 127.0.98.62:46161)
0926 20:53:10.496726 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:10.496731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:10.496766 (+ 35us) server_negotiation.cc:407] Connection header received
0926 20:53:10.497154 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:10.497156 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:10.497174 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:10.497255 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:10.497800 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:10.498595 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:10.499490 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:10.500001 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:10.500122 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:10.503001 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:10.503008 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:10.503009 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:10.503015 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:10.504797 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:10.505296 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:10.505299 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:10.505299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:10.505377 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:10.505703 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:10.505704 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:10.505705 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:10.505962 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:10.506027 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:10.506270 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 20:53:10.506376 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":88,"threads_started":1}
I20260926 20:53:10.506634 13262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:10.496390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46161 (local address 127.0.0.1:39798)
0926 20:53:10.496555 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:10.496588 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:10.497016 (+ 428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:10.497277 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:10.497279 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:10.497514 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:10.497661 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:10.497665 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:10.498681 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:10.498682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:10.499368 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:10.499370 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:10.500103 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:10.500104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:10.500193 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:10.500799 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:10.500809 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:10.502888 (+ 2079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:53:10.504882 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:10.504889 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:10.504892 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:10.505197 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:10.505444 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:10.505445 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:10.505445 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:10.505567 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:10.506033 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:53:10.506040 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:53:10.506134 (+ 94us) client_negotiation.cc:769] Sending connection context
0926 20:53:10.506244 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 20:53:10.506481 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":63,"thread_start_us":86,"threads_started":1}
I20260926 20:53:10.507496 13260 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37057 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:53:10.507906 13262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:10.507780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37057 (local address 127.0.0.1:44652)
0926 20:53:10.507836 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:10.507885 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 20:53:10.508065 13211 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37057: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260926 20:53:10.508543 13262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:10.508413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37057 (local address 127.0.0.1:44660)
0926 20:53:10.508488 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:10.508519 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 20:53:10.508669 13260 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37057: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37057: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260926 20:53:10.528903 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37057
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46161,127.0.98.61:37057,127.0.98.60:34115
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:53:10.557898 13260 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37057 } attempt: 1
I20260926 20:53:10.558419 13262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:10.558277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37057 (local address 127.0.0.1:44666)
0926 20:53:10.558350 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:10.558388 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 20:53:10.558764 13262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:10.558666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37057 (local address 127.0.0.1:44668)
0926 20:53:10.558708 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:10.558737 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 20:53:10.558862 13260 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37057: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37057: connect: Connection refused (error 111)
W20260926 20:53:10.609388 13267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:53:10.609694 13267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:53:10.609767 13267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:53:10.611002 13267 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:53:10.611068 13267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:53:10.611091 13267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:53:10.611111 13267 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:53:10.611140 13267 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:53:10.612703 13267 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46161,127.0.98.61:37057,127.0.98.60:34115
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:53:10.612975 13267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:53:10.613204 13267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:53:10.618489 13273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:10.618611 13267 server_base.cc:1034] running on GCE node
W20260926 20:53:10.618510 13272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:10.618582 13275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:10.619086 13267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:53:10.619477 13267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:53:10.620649 13267 hybrid_clock.cc:648] HybridClock initialized: now 1790455990620641 us; error 35 us; skew 500 ppm
Sep 26 20:53:10 dist-test-slave-hwlj krb5kdc[13179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:53:10.623215 13267 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260926 20:53:10.623724 13267 webserver.cc:460] Webserver started at http://127.0.98.61:41591/ using document root <none> and password file <none>
I20260926 20:53:10.624018 13267 fs_manager.cc:359] Metadata directory not provided
I20260926 20:53:10.624068 13267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:53:10.624198 13267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:53:10.625298 13267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/minicluster-data/master-1/data/instance:
uuid: "4301eb00fb2a4908a4bd22496c504ea5"
format_stamp: "Formatted at 2026-09-26 20:53:10 on dist-test-slave-hwlj"
server_key: "c9c03814b9034c37bce3cf5d04ce0d99"
server_key_iv: "0b0dbfa874a2de75277c821678714370"
server_key_version: "encryptionkey@0"
I20260926 20:53:10.625658 13267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/minicluster-data/master-1/wal/instance:
uuid: "4301eb00fb2a4908a4bd22496c504ea5"
format_stamp: "Formatted at 2026-09-26 20:53:10 on dist-test-slave-hwlj"
server_key: "c9c03814b9034c37bce3cf5d04ce0d99"
server_key_iv: "0b0dbfa874a2de75277c821678714370"
server_key_version: "encryptionkey@0"
I20260926 20:53:10.627024 13267 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:53:10.627030 13260 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37057 } attempt: 2
I20260926 20:53:10.627534 13262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:10.627399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37057 (local address 127.0.0.1:44670)
0926 20:53:10.627457 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:10.627493 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 20:53:10.627920 13262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:10.627812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37057 (local address 127.0.0.1:44678)
0926 20:53:10.627860 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:10.627895 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260926 20:53:10.628063 13260 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37057: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37057: connect: Connection refused (error 111)
I20260926 20:53:10.628134 13282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:10.628340 13267 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:53:10.628417 13267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/minicluster-data/master-1/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/minicluster-data/master-1/wal
uuid: "4301eb00fb2a4908a4bd22496c504ea5"
format_stamp: "Formatted at 2026-09-26 20:53:10 on dist-test-slave-hwlj"
server_key: "c9c03814b9034c37bce3cf5d04ce0d99"
server_key_iv: "0b0dbfa874a2de75277c821678714370"
server_key_version: "encryptionkey@0"
I20260926 20:53:10.628470 13267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:53:10.650393 13267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:53:10.652722 13267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:53:10.652841 13267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:53:10.657227 13267 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:37057
I20260926 20:53:10.657262 13334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:37057 every 8 connection(s)
I20260926 20:53:10.657647 13267 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/minicluster-data/master-1/data/info.pb
I20260926 20:53:10.658506 13335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:53:10.660071 13335 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46161 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:53:10.663945 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 13267
I20260926 20:53:10.664016 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/minicluster-data/master-1/wal/instance
I20260926 20:53:10.664177 392 external_mini_cluster.cc:1428] Setting key e3ea123e9329661d96c9e5772ee427b3
Sep 26 20:53:10 dist-test-slave-hwlj krb5kdc[13179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:53:10.669788 13263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:10.660874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39810 (local address 127.0.98.62:46161)
0926 20:53:10.660941 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:10.660943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:10.661151 (+ 208us) server_negotiation.cc:407] Connection header received
0926 20:53:10.661662 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:10.661663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:10.661680 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:10.661741 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:10.662351 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:10.662871 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:10.663935 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:10.664203 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:10.664284 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:10.666600 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:10.666601 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:10.666602 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:10.666609 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:10.668163 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:10.668669 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:10.668670 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:10.668671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:10.668726 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:10.669066 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:10.669067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:10.669067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:10.669188 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:10.669253 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:10.669641 (+ 388us) server_negotiation.cc:299] Negotiation successful
0926 20:53:10.669721 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 20:53:10.670068 13337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:10.660807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46161 (local address 127.0.0.1:39810)
0926 20:53:10.661048 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:10.661078 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:10.661549 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:10.661810 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:10.661814 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:10.662065 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:10.662259 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:10.662263 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:10.662972 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:10.662973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:10.663844 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:10.663847 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:10.664350 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:10.664351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:10.664451 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:10.664972 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:10.664981 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:10.666430 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:53:10.668243 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:10.668252 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:10.668256 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:10.668580 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:10.668800 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:10.668801 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:10.668801 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:10.668958 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:10.669267 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:53:10.669275 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:53:10.669459 (+ 184us) client_negotiation.cc:769] Sending connection context
0926 20:53:10.669593 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 20:53:10.669780 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":89,"threads_started":1}
I20260926 20:53:10.670749 13335 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37057 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 26 20:53:10 dist-test-slave-hwlj krb5kdc[13179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 20:53:10.679656 13337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:10.671084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37057 (local address 127.0.0.1:44684)
0926 20:53:10.671138 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:10.671150 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:10.671194 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:10.671549 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:10.671550 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:10.671715 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:10.671836 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:10.671838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:10.673281 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:10.673282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:10.673787 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:10.673788 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:10.674313 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:10.674314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:10.674375 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:10.674961 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:10.674965 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:10.676721 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:53:10.678616 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:10.678617 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:10.678617 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:10.678778 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:10.678993 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:10.678994 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:10.678994 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:10.679043 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:10.679366 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:53:10.679366 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:53:10.679404 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 20:53:10.679458 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 20:53:10.679587 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 20:53:10.679950 13339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:10.671163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44684 (local address 127.0.98.61:37057)
0926 20:53:10.671361 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:10.671366 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:10.671391 (+ 25us) server_negotiation.cc:407] Connection header received
0926 20:53:10.671415 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:10.671418 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:10.671432 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:10.671521 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:10.671901 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:10.673171 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:10.673864 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:10.674231 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:10.674289 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:10.676825 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:10.676835 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:10.676835 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:10.676846 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:10.678546 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:10.678860 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:10.678863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:10.678863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:10.678926 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:10.679103 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:10.679104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:10.679105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:10.679296 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:10.679379 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:10.679490 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 20:53:10.679801 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":96,"server-negotiator.queue_time_us":164,"thread_start_us":108,"threads_started":1}
I20260926 20:53:10.680909 13335 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34115 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:53:10.681293 13337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:10.681166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34115 (local address 127.0.0.1:54246)
0926 20:53:10.681220 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:10.681265 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 20:53:10.681545 13285 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34115: connect: Connection refused (error 111)
I20260926 20:53:10.682051 13337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:10.681925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34115 (local address 127.0.0.1:54254)
0926 20:53:10.681981 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:10.682016 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 20:53:10.682231 13335 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34115: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34115: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260926 20:53:10.694234 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34115
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46161,127.0.98.61:37057,127.0.98.60:34115
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:53:10.719246 13260 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37057 } attempt: 3
Sep 26 20:53:10 dist-test-slave-hwlj krb5kdc[13179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 20:53:10.727968 13262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:10.719724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37057 (local address 127.0.0.1:44694)
0926 20:53:10.719789 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:10.719803 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:10.719876 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:10.720077 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:10.720078 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:10.720248 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:10.720319 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:10.720321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:10.721284 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:10.721286 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:10.721860 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:10.721872 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:10.722403 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:10.722404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:10.722477 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:10.723068 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:10.723073 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:10.724636 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:53:10.726594 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:10.726595 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:10.726595 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:10.726818 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:10.727092 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:10.727094 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:10.727094 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:10.727163 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:10.727526 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:53:10.727527 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:53:10.727576 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 20:53:10.727738 (+ 162us) client_negotiation.cc:240] Negotiation successful
0926 20:53:10.727893 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 20:53:10.727995 13339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:10.719741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44694 (local address 127.0.98.61:37057)
0926 20:53:10.719814 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:10.719816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:10.719878 (+ 62us) server_negotiation.cc:407] Connection header received
0926 20:53:10.719952 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:10.719953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:10.719966 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:10.720025 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:10.720407 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:10.721168 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:10.721968 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:10.722311 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:10.722391 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:10.724779 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:10.724780 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:10.724780 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:10.724789 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:10.726485 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:10.726921 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:10.726922 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:10.726923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:10.726993 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:10.727277 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:10.727278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:10.727278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:10.727436 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:10.727508 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:10.727808 (+ 300us) server_negotiation.cc:299] Negotiation successful
0926 20:53:10.727903 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 20:53:10.728410 13260 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34115 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:53:10.728796 13262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:10.728681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34115 (local address 127.0.0.1:54262)
0926 20:53:10.728733 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:10.728763 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 20:53:10.729168 13262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:10.729086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34115 (local address 127.0.0.1:54272)
0926 20:53:10.729131 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:10.729154 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 20:53:10.729276 13260 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34115: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34115: connect: Connection refused (error 111)
I20260926 20:53:10.731467 13335 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34115 } attempt: 1
I20260926 20:53:10.731967 13337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:10.731837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34115 (local address 127.0.0.1:54274)
0926 20:53:10.731893 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:10.731929 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 20:53:10.732360 13337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:10.732264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34115 (local address 127.0.0.1:54282)
0926 20:53:10.732311 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:10.732336 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 20:53:10.732450 13335 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34115: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34115: connect: Connection refused (error 111)
I20260926 20:53:10.760417 13260 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34115 } attempt: 1
I20260926 20:53:10.760965 13262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:10.760825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34115 (local address 127.0.0.1:54298)
0926 20:53:10.760893 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:10.760935 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 20:53:10.761353 13262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:10.761270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34115 (local address 127.0.0.1:54300)
0926 20:53:10.761309 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:10.761337 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 20:53:10.761471 13260 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34115: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34115: connect: Connection refused (error 111)
W20260926 20:53:10.781980 13342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:53:10.782232 13342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:53:10.782269 13342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:53:10.783485 13342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:53:10.783556 13342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:53:10.783592 13342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:53:10.783643 13342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:53:10.783671 13342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:53:10.785326 13342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46161,127.0.98.61:37057,127.0.98.60:34115
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:53:10.785632 13342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:53:10.785871 13342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:53:10.790898 13348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:10.790958 13342 server_base.cc:1034] running on GCE node
W20260926 20:53:10.790906 13350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:10.790906 13347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:10.791370 13342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:53:10.791662 13342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:53:10.792809 13342 hybrid_clock.cc:648] HybridClock initialized: now 1790455990792807 us; error 40 us; skew 500 ppm
Sep 26 20:53:10 dist-test-slave-hwlj krb5kdc[13179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:53:10.795228 13342 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260926 20:53:10.795693 13342 webserver.cc:460] Webserver started at http://127.0.98.60:41335/ using document root <none> and password file <none>
I20260926 20:53:10.795941 13342 fs_manager.cc:359] Metadata directory not provided
I20260926 20:53:10.796011 13342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:53:10.796154 13342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:53:10.797202 13342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/minicluster-data/master-2/data/instance:
uuid: "529f620db0b0438085d5c575fb008911"
format_stamp: "Formatted at 2026-09-26 20:53:10 on dist-test-slave-hwlj"
server_key: "e1688b6d9a5d9eabcc8dd26b7d9ee674"
server_key_iv: "7c45baed1ce3c375a3e252db45aa2bca"
server_key_version: "encryptionkey@0"
I20260926 20:53:10.797557 13342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/minicluster-data/master-2/wal/instance:
uuid: "529f620db0b0438085d5c575fb008911"
format_stamp: "Formatted at 2026-09-26 20:53:10 on dist-test-slave-hwlj"
server_key: "e1688b6d9a5d9eabcc8dd26b7d9ee674"
server_key_iv: "7c45baed1ce3c375a3e252db45aa2bca"
server_key_version: "encryptionkey@0"
I20260926 20:53:10.798929 13342 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:53:10.799868 13357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:10.800069 13342 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:53:10.800168 13342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/minicluster-data/master-2/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/minicluster-data/master-2/wal
uuid: "529f620db0b0438085d5c575fb008911"
format_stamp: "Formatted at 2026-09-26 20:53:10 on dist-test-slave-hwlj"
server_key: "e1688b6d9a5d9eabcc8dd26b7d9ee674"
server_key_iv: "7c45baed1ce3c375a3e252db45aa2bca"
server_key_version: "encryptionkey@0"
I20260926 20:53:10.800252 13342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:53:10.800577 13335 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34115 } attempt: 2
I20260926 20:53:10.800976 13337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:10.800858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34115 (local address 127.0.0.1:54316)
0926 20:53:10.800912 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:10.800947 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 20:53:10.801309 13337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:10.801217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34115 (local address 127.0.0.1:54322)
0926 20:53:10.801260 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:10.801285 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 20:53:10.801389 13335 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34115: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34115: connect: Connection refused (error 111)
I20260926 20:53:10.820798 13342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:53:10.822839 13342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:53:10.823000 13342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:53:10.827127 13342 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:34115
I20260926 20:53:10.827158 13409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:34115 every 8 connection(s)
I20260926 20:53:10.827507 13342 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/minicluster-data/master-2/data/info.pb
I20260926 20:53:10.828406 13410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:53:10.829341 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 13342
I20260926 20:53:10.829437 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455725523895-392-0/minicluster-data/master-2/wal/instance
I20260926 20:53:10.829613 392 external_mini_cluster.cc:1428] Setting key cb42a147b077b481e6a7f84157b4cc5e
I20260926 20:53:10.830538 13410 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46161 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:53:10 dist-test-slave-hwlj krb5kdc[13179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:53:10.836614 13260 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34115 } attempt: 2
Sep 26 20:53:10 dist-test-slave-hwlj krb5kdc[13179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 26 20:53:10 dist-test-slave-hwlj krb5kdc[13179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 20:53:10.841691 13413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:10.830128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54330 (local address 127.0.98.60:34115)
0926 20:53:10.830309 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:10.830317 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:10.830343 (+ 26us) server_negotiation.cc:407] Connection header received
0926 20:53:10.830756 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:10.830763 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:10.830808 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:10.830918 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:10.831778 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:10.833004 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:10.834138 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:10.834563 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:10.834680 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:10.837640 (+ 2960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:10.837648 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:10.837648 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:10.837655 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:10.839874 (+ 2219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:10.840418 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:10.840421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:10.840422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:10.840524 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:10.840728 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:10.840729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:10.840729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:10.841027 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:10.841185 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:10.841308 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 20:53:10.841434 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":100,"threads_started":1}
I20260926 20:53:10.841722 13263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:10.831492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39826 (local address 127.0.98.62:46161)
0926 20:53:10.831542 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:10.831545 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:10.831740 (+ 195us) server_negotiation.cc:407] Connection header received
0926 20:53:10.831785 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:10.831786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:10.831798 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:10.831849 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:10.832370 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:10.833061 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:10.834313 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:10.834591 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:10.834656 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:10.837668 (+ 3012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:10.837670 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:10.837670 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:10.837680 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:10.839884 (+ 2204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:10.840391 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:10.840392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:10.840393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:10.840453 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:10.841034 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:10.841035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:10.841035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:10.841159 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:10.841226 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:10.841537 (+ 311us) server_negotiation.cc:299] Negotiation successful
0926 20:53:10.841637 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 20:53:10.842069 13414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:10.831390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46161 (local address 127.0.0.1:39826)
0926 20:53:10.831570 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:10.831618 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:10.831694 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:10.831888 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:10.831890 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:10.832133 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:10.832245 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:10.832249 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:10.833161 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:10.833162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:10.834233 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:10.834235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:10.834679 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:10.834680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:10.834753 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:10.835342 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:10.835353 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:10.837558 (+ 2205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:53:10.839952 (+ 2394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:10.839959 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:10.839962 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:10.840318 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:10.840505 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:10.840505 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:10.840506 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:10.840957 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:10.841255 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:53:10.841261 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:53:10.841310 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 20:53:10.841807 (+ 497us) client_negotiation.cc:240] Negotiation successful
0926 20:53:10.841946 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":117,"threads_started":1}
I20260926 20:53:10.842568 13410 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37057 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:53:10.845906 13262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:10.836907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34115 (local address 127.0.0.1:54332)
0926 20:53:10.836967 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:10.836978 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:10.837027 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:10.837768 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:10.837769 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:10.838010 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:10.838045 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:10.838047 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:10.838847 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:10.838848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:10.839538 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:10.839540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:10.840052 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:10.840053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:10.840112 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:10.840541 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:10.840545 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:10.841903 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:53:10.844788 (+ 2885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:10.844789 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:10.844789 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:10.844998 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:10.845244 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:10.845245 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:10.845245 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:10.845315 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:10.845591 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:53:10.845592 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:53:10.845636 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 20:53:10.845726 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 20:53:10.845817 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 20:53:10.845925 13415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:10.837085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54332 (local address 127.0.98.60:34115)
0926 20:53:10.837222 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:10.837224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:10.837231 (+ 7us) server_negotiation.cc:407] Connection header received
0926 20:53:10.837251 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:10.837251 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:10.837655 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:10.837747 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:10.838123 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:10.838766 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:10.839635 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:10.839978 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:10.840048 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:10.841991 (+ 1943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:10.841993 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:10.841993 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:10.841999 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:10.844721 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:10.845091 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:10.845093 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:10.845093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:10.845163 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:10.845391 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:10.845392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:10.845392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:10.845536 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:10.845598 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:10.845750 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 20:53:10.845817 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
Sep 26 20:53:10 dist-test-slave-hwlj krb5kdc[13179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 26 20:53:10 dist-test-slave-hwlj krb5kdc[13179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:53:10.848358 13260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b7a6988af6740fca5e8e04928c67ee1: Bootstrap starting.
I20260926 20:53:10.849488 13260 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b7a6988af6740fca5e8e04928c67ee1: Neither blocks nor log segments found. Creating new log.
I20260926 20:53:10.849866 13260 log.cc:824] T 00000000000000000000000000000000 P 3b7a6988af6740fca5e8e04928c67ee1: Log is configured to *not* fsync() on all Append() calls
I20260926 20:53:10.850580 13414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:10.842830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37057 (local address 127.0.0.1:44710)
0926 20:53:10.842869 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:10.842878 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:10.842907 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:10.843031 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:10.843032 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:10.843167 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:10.843215 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:10.843217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:10.844111 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:10.844112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:10.844657 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:10.844659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:10.845144 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:10.845146 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:10.845226 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:10.846098 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:10.846103 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:10.847271 (+ 1168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:53:10.849071 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:10.849072 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:10.849072 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:10.849306 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:10.849538 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:10.849539 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:10.849539 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:10.849600 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:10.850250 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:53:10.850251 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:53:10.850298 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 20:53:10.850400 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 20:53:10.850492 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260926 20:53:10.850621 13339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:10.842895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44710 (local address 127.0.98.61:37057)
0926 20:53:10.842940 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:10.842943 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:10.842950 (+ 7us) server_negotiation.cc:407] Connection header received
0926 20:53:10.842974 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:10.842975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:10.842983 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:10.843031 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:10.843293 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:10.844026 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:10.844739 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:10.845025 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:10.845087 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:10.847386 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:10.847387 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:10.847387 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:10.847394 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:10.848985 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:10.849392 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:10.849393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:10.849394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:10.849457 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:10.849678 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:10.849679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:10.849679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:10.849831 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:10.849906 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:10.850437 (+ 531us) server_negotiation.cc:299] Negotiation successful
0926 20:53:10.850529 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 20:53:10.850749 13260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b7a6988af6740fca5e8e04928c67ee1: No bootstrap required, opened a new log
I20260926 20:53:10.851140 13263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:10.842722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39838 (local address 127.0.98.62:46161)
0926 20:53:10.842756 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:10.842757 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:10.842765 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:53:10.842783 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:10.842784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:10.842793 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:10.842852 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:10.843508 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:10.844213 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:10.845271 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:10.845577 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:10.845656 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:10.848332 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:10.848333 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:10.848334 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:10.848341 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:10.849840 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:10.850239 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:10.850240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:10.850240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:10.850297 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:10.850533 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:10.850534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:10.850534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:10.850654 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:10.850779 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:10.850982 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 20:53:10.851059 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 20:53:10.851229 13410 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34115 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:53:10.852841 13260 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b7a6988af6740fca5e8e04928c67ee1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b7a6988af6740fca5e8e04928c67ee1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46161 } } peers { permanent_uuid: "4301eb00fb2a4908a4bd22496c504ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37057 } } peers { permanent_uuid: "529f620db0b0438085d5c575fb008911" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34115 } }
I20260926 20:53:10.853016 13260 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b7a6988af6740fca5e8e04928c67ee1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:53:10.853070 13260 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b7a6988af6740fca5e8e04928c67ee1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b7a6988af6740fca5e8e04928c67ee1, State: Initialized, Role: FOLLOWER
I20260926 20:53:10.853209 13260 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b7a6988af6740fca5e8e04928c67ee1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b7a6988af6740fca5e8e04928c67ee1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46161 } } peers { permanent_uuid: "4301eb00fb2a4908a4bd22496c504ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37057 } } peers { permanent_uuid: "529f620db0b0438085d5c575fb008911" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34115 } }
I20260926 20:53:10.853492 13417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b7a6988af6740fca5e8e04928c67ee1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b7a6988af6740fca5e8e04928c67ee1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46161 } } peers { permanent_uuid: "4301eb00fb2a4908a4bd22496c504ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37057 } } peers { permanent_uuid: "529f620db0b0438085d5c575fb008911" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34115 } } }
I20260926 20:53:10.853549 13260 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b7a6988af6740fca5e8e04928c67ee1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:53:10.853636 13417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b7a6988af6740fca5e8e04928c67ee1 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 20:53:10.855392 13428 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3b7a6988af6740fca5e8e04928c67ee1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:53:10.855499 13428 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 26 20:53:10 dist-test-slave-hwlj krb5kdc[13179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 26 20:53:10 dist-test-slave-hwlj krb5kdc[13179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 20:53:10.856149 13192 master_runner.cc:186] Error getting master registration for 127.0.98.61:37057: OK, instance_id { permanent_uuid: "4301eb00fb2a4908a4bd22496c504ea5" instance_seqno: 1790455990656061 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 20:53:10.858927 13192 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 20:53:10.859038 13415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:10.851590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54334 (local address 127.0.98.60:34115)
0926 20:53:10.851656 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:10.851658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:10.851666 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:53:10.851685 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:10.851685 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:10.851696 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:10.851791 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:10.852072 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:10.852754 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:10.853611 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:10.853924 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:10.853959 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:10.856108 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:10.856109 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:10.856110 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:10.856116 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:10.857794 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:10.858155 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:10.858156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:10.858156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:10.858199 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:10.858386 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:10.858386 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:10.858387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:10.858576 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:10.858621 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:10.858829 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 20:53:10.858917 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 20:53:10.859340 13414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:10.851493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34115 (local address 127.0.0.1:54334)
0926 20:53:10.851559 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:10.851569 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:10.851604 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:10.851827 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:10.851828 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:10.851970 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:10.852000 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:10.852002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:10.852832 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:10.852833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:10.853523 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:10.853525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:10.854019 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:10.854020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:10.854072 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:10.854488 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:10.854493 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:10.856009 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:53:10.857881 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:10.857881 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:10.857882 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:10.858086 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:10.858264 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:10.858264 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:10.858265 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:10.858319 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:10.858666 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:53:10.858666 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:53:10.858710 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 20:53:10.859153 (+ 443us) client_negotiation.cc:240] Negotiation successful
0926 20:53:10.859247 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 20:53:10.860908 13410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 529f620db0b0438085d5c575fb008911: Bootstrap starting.
I20260926 20:53:10.861060 13339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:10.851790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44718 (local address 127.0.98.61:37057)
0926 20:53:10.851881 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:10.851883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:10.851889 (+ 6us) server_negotiation.cc:407] Connection header received
0926 20:53:10.851904 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:10.851905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:10.851912 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:10.851977 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:10.852541 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:10.853195 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:10.854107 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:10.854697 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:10.854742 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:10.858092 (+ 3350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:10.858093 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:10.858094 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:10.858100 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:10.859855 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:10.860216 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:10.860217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:10.860218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:10.860282 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:10.860517 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:10.860518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:10.860519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:10.860648 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:10.860813 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:10.860893 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 20:53:10.860972 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 20:53:10.861652 13410 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 529f620db0b0438085d5c575fb008911: Neither blocks nor log segments found. Creating new log.
I20260926 20:53:10.861967 13410 log.cc:824] T 00000000000000000000000000000000 P 529f620db0b0438085d5c575fb008911: Log is configured to *not* fsync() on all Append() calls
I20260926 20:53:10.862351 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:53:10.862701 13410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 529f620db0b0438085d5c575fb008911: No bootstrap required, opened a new log
I20260926 20:53:10.864727 13410 raft_consensus.cc:348] T 00000000000000000000000000000000 P 529f620db0b0438085d5c575fb008911 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b7a6988af6740fca5e8e04928c67ee1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46161 } } peers { permanent_uuid: "4301eb00fb2a4908a4bd22496c504ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37057 } } peers { permanent_uuid: "529f620db0b0438085d5c575fb008911" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34115 } }
I20260926 20:53:10.864965 13410 raft_consensus.cc:374] T 00000000000000000000000000000000 P 529f620db0b0438085d5c575fb008911 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:53:10.865060 13410 raft_consensus.cc:729] T 00000000000000000000000000000000 P 529f620db0b0438085d5c575fb008911 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 529f620db0b0438085d5c575fb008911, State: Initialized, Role: FOLLOWER
I20260926 20:53:10.865243 13410 consensus_queue.cc:260] T 00000000000000000000000000000000 P 529f620db0b0438085d5c575fb008911 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b7a6988af6740fca5e8e04928c67ee1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46161 } } peers { permanent_uuid: "4301eb00fb2a4908a4bd22496c504ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37057 } } peers { permanent_uuid: "529f620db0b0438085d5c575fb008911" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34115 } }
I20260926 20:53:10.865623 13410 sys_catalog.cc:564] T 00000000000000000000000000000000 P 529f620db0b0438085d5c575fb008911 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:53:10.867290 13437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 529f620db0b0438085d5c575fb008911 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b7a6988af6740fca5e8e04928c67ee1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46161 } } peers { permanent_uuid: "4301eb00fb2a4908a4bd22496c504ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37057 } } peers { permanent_uuid: "529f620db0b0438085d5c575fb008911" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34115 } } }
I20260926 20:53:10.867436 13437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 529f620db0b0438085d5c575fb008911 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 20:53:10.868578 13448 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 529f620db0b0438085d5c575fb008911: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:53:10.868685 13448 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:53:10.869105 13342 master_runner.cc:186] Error getting master registration for 127.0.98.61:37057: OK, instance_id { permanent_uuid: "4301eb00fb2a4908a4bd22496c504ea5" instance_seqno: 1790455990656061 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 20:53:10.871912 13342 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 20:53:10.873026 13339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:10.864242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44722 (local address 127.0.98.61:37057)
0926 20:53:10.864277 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:10.864278 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:10.864504 (+ 226us) server_negotiation.cc:407] Connection header received
0926 20:53:10.864593 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:10.864594 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:10.864602 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:10.864629 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:10.865383 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:10.866088 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:10.867223 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:10.867510 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:10.867581 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:10.869520 (+ 1939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:10.869522 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:10.869522 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:10.869529 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:10.871631 (+ 2102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:10.872097 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:10.872099 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:10.872100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:10.872154 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:10.872380 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:10.872381 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:10.872381 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:10.872518 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:10.872613 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:10.872828 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 20:53:10.872911 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 20:53:10.873800 13415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:10.863909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54344 (local address 127.0.98.60:34115)
0926 20:53:10.863950 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:10.863952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:10.864234 (+ 282us) server_negotiation.cc:407] Connection header received
0926 20:53:10.864330 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:10.864332 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:10.864341 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:10.864368 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:10.865116 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:10.865835 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:10.866981 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:10.867275 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:10.867317 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:10.870289 (+ 2972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:10.870291 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:10.870291 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:10.870298 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:10.871919 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:10.872480 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:10.872481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:10.872482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:10.872533 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:10.872742 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:10.872743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:10.872743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:10.872880 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:10.872980 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:10.873625 (+ 645us) server_negotiation.cc:299] Negotiation successful
0926 20:53:10.873700 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 20:53:10.879587 13263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:10.872009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39840 (local address 127.0.98.62:46161)
0926 20:53:10.872085 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:10.872087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:10.872430 (+ 343us) server_negotiation.cc:407] Connection header received
0926 20:53:10.872522 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:10.872523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:10.872533 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:10.872562 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:10.873139 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:10.873882 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:10.875076 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:10.875262 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:10.875320 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:10.876672 (+ 1352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:10.876674 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:10.876674 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:10.876682 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:10.878275 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:10.878645 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:10.878646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:10.878646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:10.878707 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:10.878907 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:10.878908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:10.878908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:10.879035 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:10.879123 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:10.879395 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 20:53:10.879492 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 20:53:10.892535 13335 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34115 } attempt: 3
Sep 26 20:53:10 dist-test-slave-hwlj krb5kdc[13179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 20:53:10.901373 13337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:10.892840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34115 (local address 127.0.0.1:54356)
0926 20:53:10.892891 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:10.892903 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:10.892940 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:10.893097 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:10.893098 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:10.893293 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:10.893344 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:10.893345 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:10.894283 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:10.894284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:10.895096 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:10.895098 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:10.896019 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:10.896020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:10.896086 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:10.896689 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:10.896694 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:10.898291 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:53:10.900216 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:10.900217 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:10.900217 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:10.900469 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:10.900696 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:10.900697 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:10.900697 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:10.900766 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:10.901048 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:53:10.901049 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:53:10.901097 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 20:53:10.901169 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 20:53:10.901267 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 20:53:10.901839 13415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:10.892903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54356 (local address 127.0.98.60:34115)
0926 20:53:10.892945 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:10.892947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:10.892981 (+ 34us) server_negotiation.cc:407] Connection header received
0926 20:53:10.893004 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:10.893005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:10.893015 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:10.893067 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:10.893429 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:10.894186 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:10.895184 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:10.895515 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:10.895714 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:10.898411 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:10.898413 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:10.898413 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:10.898424 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:10.900128 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:10.900546 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:10.900547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:10.900547 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:10.900615 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:10.900837 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:10.900838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:10.900838 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:10.900977 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:10.901046 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:10.901208 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 20:53:10.901297 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 20:53:10.904135 13335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4301eb00fb2a4908a4bd22496c504ea5: Bootstrap starting.
I20260926 20:53:10.904925 13335 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4301eb00fb2a4908a4bd22496c504ea5: Neither blocks nor log segments found. Creating new log.
I20260926 20:53:10.905274 13335 log.cc:824] T 00000000000000000000000000000000 P 4301eb00fb2a4908a4bd22496c504ea5: Log is configured to *not* fsync() on all Append() calls
I20260926 20:53:10.906188 13335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4301eb00fb2a4908a4bd22496c504ea5: No bootstrap required, opened a new log
I20260926 20:53:10.908205 13335 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4301eb00fb2a4908a4bd22496c504ea5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b7a6988af6740fca5e8e04928c67ee1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46161 } } peers { permanent_uuid: "4301eb00fb2a4908a4bd22496c504ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37057 } } peers { permanent_uuid: "529f620db0b0438085d5c575fb008911" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34115 } }
I20260926 20:53:10.908366 13335 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4301eb00fb2a4908a4bd22496c504ea5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:53:10.908399 13335 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4301eb00fb2a4908a4bd22496c504ea5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4301eb00fb2a4908a4bd22496c504ea5, State: Initialized, Role: FOLLOWER
I20260926 20:53:10.908573 13335 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4301eb00fb2a4908a4bd22496c504ea5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b7a6988af6740fca5e8e04928c67ee1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46161 } } peers { permanent_uuid: "4301eb00fb2a4908a4bd22496c504ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37057 } } peers { permanent_uuid: "529f620db0b0438085d5c575fb008911" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34115 } }
I20260926 20:53:10.908928 13335 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4301eb00fb2a4908a4bd22496c504ea5 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 20:53:10.911070 13463 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4301eb00fb2a4908a4bd22496c504ea5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:53:10.911178 13463 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:53:10.912945 13267 master_runner.cc:263] No leader master found from master 4301eb00fb2a4908a4bd22496c504ea5
I20260926 20:53:10.912993 13267 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 20:53:10.908891 13452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4301eb00fb2a4908a4bd22496c504ea5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b7a6988af6740fca5e8e04928c67ee1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46161 } } peers { permanent_uuid: "4301eb00fb2a4908a4bd22496c504ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37057 } } peers { permanent_uuid: "529f620db0b0438085d5c575fb008911" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34115 } } }
I20260926 20:53:10.913187 13452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4301eb00fb2a4908a4bd22496c504ea5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:53:10.954649 13452 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4301eb00fb2a4908a4bd22496c504ea5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:53:10.954813 13452 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4301eb00fb2a4908a4bd22496c504ea5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b7a6988af6740fca5e8e04928c67ee1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46161 } } peers { permanent_uuid: "4301eb00fb2a4908a4bd22496c504ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37057 } } peers { permanent_uuid: "529f620db0b0438085d5c575fb008911" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34115 } }
I20260926 20:53:10.955556 13452 leader_election.cc:288] T 00000000000000000000000000000000 P 4301eb00fb2a4908a4bd22496c504ea5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b7a6988af6740fca5e8e04928c67ee1 (127.0.98.62:46161), 529f620db0b0438085d5c575fb008911 (127.0.98.60:34115)
I20260926 20:53:10.956038 13384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4301eb00fb2a4908a4bd22496c504ea5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "529f620db0b0438085d5c575fb008911" is_pre_election: true
I20260926 20:53:10.956218 13384 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 529f620db0b0438085d5c575fb008911 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4301eb00fb2a4908a4bd22496c504ea5 in term 0.
I20260926 20:53:10.956547 13285 leader_election.cc:302] T 00000000000000000000000000000000 P 4301eb00fb2a4908a4bd22496c504ea5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4301eb00fb2a4908a4bd22496c504ea5, 529f620db0b0438085d5c575fb008911; no voters:
I20260926 20:53:10.956943 13234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4301eb00fb2a4908a4bd22496c504ea5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b7a6988af6740fca5e8e04928c67ee1" is_pre_election: true
I20260926 20:53:10.957089 13234 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3b7a6988af6740fca5e8e04928c67ee1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4301eb00fb2a4908a4bd22496c504ea5 in term 0.
I20260926 20:53:10.957415 13452 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4301eb00fb2a4908a4bd22496c504ea5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:53:10.957474 13452 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4301eb00fb2a4908a4bd22496c504ea5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:53:10.957499 13452 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4301eb00fb2a4908a4bd22496c504ea5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:53:10.958428 13452 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4301eb00fb2a4908a4bd22496c504ea5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b7a6988af6740fca5e8e04928c67ee1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46161 } } peers { permanent_uuid: "4301eb00fb2a4908a4bd22496c504ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37057 } } peers { permanent_uuid: "529f620db0b0438085d5c575fb008911" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34115 } }
I20260926 20:53:10.958750 13234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4301eb00fb2a4908a4bd22496c504ea5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b7a6988af6740fca5e8e04928c67ee1"
I20260926 20:53:10.958827 13234 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b7a6988af6740fca5e8e04928c67ee1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:53:10.959723 13234 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3b7a6988af6740fca5e8e04928c67ee1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4301eb00fb2a4908a4bd22496c504ea5 in term 1.
I20260926 20:53:10.960017 13284 leader_election.cc:302] T 00000000000000000000000000000000 P 4301eb00fb2a4908a4bd22496c504ea5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b7a6988af6740fca5e8e04928c67ee1, 4301eb00fb2a4908a4bd22496c504ea5; no voters:
I20260926 20:53:10.960407 13384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4301eb00fb2a4908a4bd22496c504ea5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "529f620db0b0438085d5c575fb008911"
I20260926 20:53:10.960513 13384 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 529f620db0b0438085d5c575fb008911 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:53:10.961311 13384 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 529f620db0b0438085d5c575fb008911 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4301eb00fb2a4908a4bd22496c504ea5 in term 1.
I20260926 20:53:10.961551 13452 leader_election.cc:288] T 00000000000000000000000000000000 P 4301eb00fb2a4908a4bd22496c504ea5 [CANDIDATE]: Term 1 election: Requested vote from peers 3b7a6988af6740fca5e8e04928c67ee1 (127.0.98.62:46161), 529f620db0b0438085d5c575fb008911 (127.0.98.60:34115)
I20260926 20:53:10.961661 13452 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4301eb00fb2a4908a4bd22496c504ea5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:53:10.961743 13452 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4301eb00fb2a4908a4bd22496c504ea5 [term 1 LEADER]: Becoming Leader. State: Replica: 4301eb00fb2a4908a4bd22496c504ea5, State: Running, Role: LEADER
I20260926 20:53:10.961885 13452 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4301eb00fb2a4908a4bd22496c504ea5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b7a6988af6740fca5e8e04928c67ee1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46161 } } peers { permanent_uuid: "4301eb00fb2a4908a4bd22496c504ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37057 } } peers { permanent_uuid: "529f620db0b0438085d5c575fb008911" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34115 } }
I20260926 20:53:10.962472 13452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4301eb00fb2a4908a4bd22496c504ea5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4301eb00fb2a4908a4bd22496c504ea5. Latest consensus state: current_term: 1 leader_uuid: "4301eb00fb2a4908a4bd22496c504ea5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b7a6988af6740fca5e8e04928c67ee1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46161 } } peers { permanent_uuid: "4301eb00fb2a4908a4bd22496c504ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37057 } } peers { permanent_uuid: "529f620db0b0438085d5c575fb008911" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34115 } } }
I20260926 20:53:10.962661 13452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4301eb00fb2a4908a4bd22496c504ea5 [sys.catalog]: This master's current role is: LEADER
I20260926 20:53:10.962947 13467 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:53:10.963193 13467 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:53:10.965577 13384 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 529f620db0b0438085d5c575fb008911 [term 1 FOLLOWER]: Refusing update from remote peer 4301eb00fb2a4908a4bd22496c504ea5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:53:10.967027 13465 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4301eb00fb2a4908a4bd22496c504ea5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "529f620db0b0438085d5c575fb008911" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:53:10.968501 13437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 529f620db0b0438085d5c575fb008911 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4301eb00fb2a4908a4bd22496c504ea5. Latest consensus state: current_term: 1 leader_uuid: "4301eb00fb2a4908a4bd22496c504ea5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b7a6988af6740fca5e8e04928c67ee1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46161 } } peers { permanent_uuid: "4301eb00fb2a4908a4bd22496c504ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37057 } } peers { permanent_uuid: "529f620db0b0438085d5c575fb008911" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34115 } } }
I20260926 20:53:10.968588 13437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 529f620db0b0438085d5c575fb008911 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:53:10.969034 13234 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3b7a6988af6740fca5e8e04928c67ee1 [term 1 FOLLOWER]: Refusing update from remote peer 4301eb00fb2a4908a4bd22496c504ea5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:53:10.969373 13465 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4301eb00fb2a4908a4bd22496c504ea5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b7a6988af6740fca5e8e04928c67ee1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:53:10.970593 13417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b7a6988af6740fca5e8e04928c67ee1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4301eb00fb2a4908a4bd22496c504ea5. Latest consensus state: current_term: 1 leader_uuid: "4301eb00fb2a4908a4bd22496c504ea5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b7a6988af6740fca5e8e04928c67ee1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46161 } } peers { permanent_uuid: "4301eb00fb2a4908a4bd22496c504ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37057 } } peers { permanent_uuid: "529f620db0b0438085d5c575fb008911" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34115 } } }
I20260926 20:53:10.970687 13417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b7a6988af6740fca5e8e04928c67ee1 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:53:10.971719 13465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4301eb00fb2a4908a4bd22496c504ea5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4301eb00fb2a4908a4bd22496c504ea5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b7a6988af6740fca5e8e04928c67ee1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46161 } } peers { permanent_uuid: "4301eb00fb2a4908a4bd22496c504ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37057 } } peers { permanent_uuid: "529f620db0b0438085d5c575fb008911" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34115 } } }
I20260926 20:53:10.971848 13465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4301eb00fb2a4908a4bd22496c504ea5 [sys.catalog]: This master's current role is: LEADER
I20260926 20:53:10.972225 13417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b7a6988af6740fca5e8e04928c67ee1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4301eb00fb2a4908a4bd22496c504ea5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b7a6988af6740fca5e8e04928c67ee1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46161 } } peers { permanent_uuid: "4301eb00fb2a4908a4bd22496c504ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37057 } } peers { permanent_uuid: "529f620db0b0438085d5c575fb008911" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34115 } } }
I20260926 20:53:10.972312 13417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b7a6988af6740fca5e8e04928c67ee1 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:53:10.972616 13468 mvcc.cc:205] Tried to move back new op lower bound from 7333707738992115712 to 7333707738981044224. Current Snapshot: MvccSnapshot[applied={T|T < 7333707738992115712}]
I20260926 20:53:10.973155 13471 mvcc.cc:205] Tried to move back new op lower bound from 7333707738992115712 to 7333707738981044224. Current Snapshot: MvccSnapshot[applied={T|T < 7333707738992115712}]
I20260926 20:53:10.973989 13452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4301eb00fb2a4908a4bd22496c504ea5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4301eb00fb2a4908a4bd22496c504ea5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b7a6988af6740fca5e8e04928c67ee1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46161 } } peers { permanent_uuid: "4301eb00fb2a4908a4bd22496c504ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37057 } } peers { permanent_uuid: "529f620db0b0438085d5c575fb008911" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34115 } } }
I20260926 20:53:10.974066 13452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4301eb00fb2a4908a4bd22496c504ea5 [sys.catalog]: This master's current role is: LEADER
I20260926 20:53:10.974454 13437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 529f620db0b0438085d5c575fb008911 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4301eb00fb2a4908a4bd22496c504ea5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b7a6988af6740fca5e8e04928c67ee1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46161 } } peers { permanent_uuid: "4301eb00fb2a4908a4bd22496c504ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37057 } } peers { permanent_uuid: "529f620db0b0438085d5c575fb008911" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34115 } } }
I20260926 20:53:10.974541 13437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 529f620db0b0438085d5c575fb008911 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:53:10.975615 13469 mvcc.cc:205] Tried to move back new op lower bound from 7333707738992115712 to 7333707738981044224. Current Snapshot: MvccSnapshot[applied={T|T < 7333707738992115712}]
I20260926 20:53:10.975802 13467 catalog_manager.cc:1347] Generated new cluster ID: d5b44f82626d410c876762d5891d2312
I20260926 20:53:10.975850 13467 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:53:11.004379 13467 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:53:11.005209 13467 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:53:11.010596 13467 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4301eb00fb2a4908a4bd22496c504ea5: Generated new TSK 0
I20260926 20:53:11.010860 13467 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:53:11.856011 13428 catalog_manager.cc:1259] Loaded cluster ID: d5b44f82626d410c876762d5891d2312
I20260926 20:53:11.856149 13428 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3b7a6988af6740fca5e8e04928c67ee1: loading cluster ID for follower catalog manager: success
I20260926 20:53:11.857077 13428 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3b7a6988af6740fca5e8e04928c67ee1: acquiring CA information for follower catalog manager: success
I20260926 20:53:11.857434 13428 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3b7a6988af6740fca5e8e04928c67ee1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 20:53:11.869206 13448 catalog_manager.cc:1259] Loaded cluster ID: d5b44f82626d410c876762d5891d2312
I20260926 20:53:11.869278 13448 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 529f620db0b0438085d5c575fb008911: loading cluster ID for follower catalog manager: success
I20260926 20:53:11.870050 13448 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 529f620db0b0438085d5c575fb008911: acquiring CA information for follower catalog manager: success
I20260926 20:53:11.870321 13448 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 529f620db0b0438085d5c575fb008911: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 20:53:12.069964 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 13192
I20260926 20:53:12.075335 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 13267
I20260926 20:53:12.080120 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 13342
2026-09-26T20:53:12Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2874 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455725523895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:53:12 dist-test-slave-hwlj krb5kdc[13481](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:53:12 dist-test-slave-hwlj krb5kdc[13481](info): set up 2 sockets
Sep 26 20:53:12 dist-test-slave-hwlj krb5kdc[13481](info): commencing operation
krb5kdc: starting...
W20260926 20:53:13.119717 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455725523895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455725523895-392-0/krb5kdc/test-user.keytab.
Sep 26 20:53:13 dist-test-slave-hwlj krb5kdc[13481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:53:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:53:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 20:53:13.243948 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34527
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34527
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:53:13.329393 13494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:53:13.329682 13494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:53:13.329737 13494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:53:13.330955 13494 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:53:13.331030 13494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:53:13.331054 13494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:53:13.331074 13494 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:53:13.331105 13494 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:53:13.332688 13494 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455725523895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:53:13.332973 13494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:53:13.333211 13494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:53:13.337936 13499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:13.338061 13502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:13.338099 13500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:13.338490 13494 server_base.cc:1034] running on GCE node
I20260926 20:53:13.338691 13494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:53:13.338933 13494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:53:13.340068 13494 hybrid_clock.cc:648] HybridClock initialized: now 1790455993340051 us; error 33 us; skew 500 ppm
Sep 26 20:53:13 dist-test-slave-hwlj krb5kdc[13481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:53:13.342712 13494 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:53:13.343202 13494 webserver.cc:460] Webserver started at http://127.0.98.62:38207/ using document root <none> and password file <none>
I20260926 20:53:13.343449 13494 fs_manager.cc:359] Metadata directory not provided
I20260926 20:53:13.343495 13494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:53:13.343708 13494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:53:13.344816 13494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455725523895-392-0/minicluster-data/master-0/data/instance:
uuid: "94962161bfbf49f6b696e7df108514a4"
format_stamp: "Formatted at 2026-09-26 20:53:13 on dist-test-slave-hwlj"
server_key: "326dddac69bec620e684d4b9bd52c767"
server_key_iv: "98aecaa4955472695bf2ad3fdd93c3fa"
server_key_version: "encryptionkey@0"
I20260926 20:53:13.345176 13494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455725523895-392-0/minicluster-data/master-0/wal/instance:
uuid: "94962161bfbf49f6b696e7df108514a4"
format_stamp: "Formatted at 2026-09-26 20:53:13 on dist-test-slave-hwlj"
server_key: "326dddac69bec620e684d4b9bd52c767"
server_key_iv: "98aecaa4955472695bf2ad3fdd93c3fa"
server_key_version: "encryptionkey@0"
I20260926 20:53:13.346535 13494 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:53:13.347297 13509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:13.347484 13494 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:53:13.347575 13494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "94962161bfbf49f6b696e7df108514a4"
format_stamp: "Formatted at 2026-09-26 20:53:13 on dist-test-slave-hwlj"
server_key: "326dddac69bec620e684d4b9bd52c767"
server_key_iv: "98aecaa4955472695bf2ad3fdd93c3fa"
server_key_version: "encryptionkey@0"
I20260926 20:53:13.347674 13494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:53:13.378378 13494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:53:13.380511 13494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:53:13.380689 13494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:53:13.384778 13494 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34527
I20260926 20:53:13.384820 13561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34527 every 8 connection(s)
I20260926 20:53:13.385215 13494 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:53:13.386075 13562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:53:13.388484 13562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94962161bfbf49f6b696e7df108514a4: Bootstrap starting.
I20260926 20:53:13.389036 13562 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94962161bfbf49f6b696e7df108514a4: Neither blocks nor log segments found. Creating new log.
I20260926 20:53:13.389225 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 13494
I20260926 20:53:13.389329 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455725523895-392-0/minicluster-data/master-0/wal/instance
I20260926 20:53:13.389381 13562 log.cc:824] T 00000000000000000000000000000000 P 94962161bfbf49f6b696e7df108514a4: Log is configured to *not* fsync() on all Append() calls
I20260926 20:53:13.389549 392 external_mini_cluster.cc:1428] Setting key 1847f7864394ec0accaefe939778ed4d
I20260926 20:53:13.390201 13562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94962161bfbf49f6b696e7df108514a4: No bootstrap required, opened a new log
I20260926 20:53:13.392477 13562 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94962161bfbf49f6b696e7df108514a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94962161bfbf49f6b696e7df108514a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34527 } }
I20260926 20:53:13.392642 13562 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94962161bfbf49f6b696e7df108514a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:53:13.392700 13562 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94962161bfbf49f6b696e7df108514a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94962161bfbf49f6b696e7df108514a4, State: Initialized, Role: FOLLOWER
I20260926 20:53:13.392834 13562 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94962161bfbf49f6b696e7df108514a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94962161bfbf49f6b696e7df108514a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34527 } }
I20260926 20:53:13.392927 13562 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94962161bfbf49f6b696e7df108514a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:53:13.392971 13562 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94962161bfbf49f6b696e7df108514a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:53:13.393020 13562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94962161bfbf49f6b696e7df108514a4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:53:13.393575 13562 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94962161bfbf49f6b696e7df108514a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94962161bfbf49f6b696e7df108514a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34527 } }
I20260926 20:53:13.393708 13562 leader_election.cc:302] T 00000000000000000000000000000000 P 94962161bfbf49f6b696e7df108514a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94962161bfbf49f6b696e7df108514a4; no voters:
I20260926 20:53:13.393915 13562 leader_election.cc:288] T 00000000000000000000000000000000 P 94962161bfbf49f6b696e7df108514a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:53:13.393940 13567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94962161bfbf49f6b696e7df108514a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:53:13.394053 13567 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94962161bfbf49f6b696e7df108514a4 [term 1 LEADER]: Becoming Leader. State: Replica: 94962161bfbf49f6b696e7df108514a4, State: Running, Role: LEADER
I20260926 20:53:13.394201 13562 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94962161bfbf49f6b696e7df108514a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:53:13.394204 13567 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94962161bfbf49f6b696e7df108514a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94962161bfbf49f6b696e7df108514a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34527 } }
I20260926 20:53:13.394882 13568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94962161bfbf49f6b696e7df108514a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94962161bfbf49f6b696e7df108514a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94962161bfbf49f6b696e7df108514a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34527 } } }
I20260926 20:53:13.395040 13569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94962161bfbf49f6b696e7df108514a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94962161bfbf49f6b696e7df108514a4. Latest consensus state: current_term: 1 leader_uuid: "94962161bfbf49f6b696e7df108514a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94962161bfbf49f6b696e7df108514a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34527 } } }
I20260926 20:53:13.395107 13568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94962161bfbf49f6b696e7df108514a4 [sys.catalog]: This master's current role is: LEADER
I20260926 20:53:13.395175 13569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94962161bfbf49f6b696e7df108514a4 [sys.catalog]: This master's current role is: LEADER
I20260926 20:53:13.395469 13577 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:53:13.396034 13577 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 20:53:13 dist-test-slave-hwlj krb5kdc[13481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:53:13.397679 13577 catalog_manager.cc:1347] Generated new cluster ID: 95686b41ce6943d9834d039be00af5e3
I20260926 20:53:13.397732 13577 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:53:13.403091 13566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:13.390055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39054 (local address 127.0.98.62:34527)
0926 20:53:13.390222 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:13.390233 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:13.390270 (+ 37us) server_negotiation.cc:407] Connection header received
0926 20:53:13.390720 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:13.390727 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:13.390803 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:13.390956 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:13.391594 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:13.392542 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:13.393466 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:13.393995 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:13.394105 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:13.397200 (+ 3095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:13.397204 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:13.397205 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:13.397218 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:13.401361 (+ 4143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:13.401842 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:13.401845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:13.401846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:13.401951 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:13.402173 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:13.402174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:13.402174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:13.402462 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:13.402620 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:13.402750 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 20:53:13.402884 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":81,"threads_started":1}
I20260926 20:53:13.404204 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:53:13.405133 13577 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:53:13.405632 13577 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:53:13.413053 13577 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94962161bfbf49f6b696e7df108514a4: Generated new TSK 0
I20260926 20:53:13.413287 13577 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:53:13.458417 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 13494
2026-09-26T20:53:13Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1376 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:53:13 dist-test-slave-hwlj krb5kdc[13589](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:53:13 dist-test-slave-hwlj krb5kdc[13589](info): set up 2 sockets
Sep 26 20:53:13 dist-test-slave-hwlj krb5kdc[13589](info): commencing operation
krb5kdc: starting...
W20260926 20:53:14.487615 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/test-user.keytab.
Sep 26 20:53:14 dist-test-slave-hwlj krb5kdc[13589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:53:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:53:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 20:53:14.611526 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40317
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40317
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:53:14.697937 13602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:53:14.698191 13602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:53:14.698232 13602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:53:14.699565 13602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:53:14.699643 13602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:53:14.699671 13602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:53:14.699703 13602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:53:14.699728 13602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:53:14.701330 13602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:53:14.701637 13602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:53:14.701876 13602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:53:14.707039 13602 server_base.cc:1034] running on GCE node
W20260926 20:53:14.707010 13607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:14.707010 13608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:14.707036 13610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:14.707527 13602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:53:14.707932 13602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:53:14.709131 13602 hybrid_clock.cc:648] HybridClock initialized: now 1790455994709121 us; error 40 us; skew 500 ppm
Sep 26 20:53:14 dist-test-slave-hwlj krb5kdc[13589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455994, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:53:14.711826 13602 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:53:14.712257 13602 webserver.cc:460] Webserver started at http://127.0.98.62:42341/ using document root <none> and password file <none>
I20260926 20:53:14.712514 13602 fs_manager.cc:359] Metadata directory not provided
I20260926 20:53:14.712568 13602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:53:14.712716 13602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:53:14.713886 13602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/master-0/data/instance:
uuid: "c34ec07649dc4073aacd19b4012322dc"
format_stamp: "Formatted at 2026-09-26 20:53:14 on dist-test-slave-hwlj"
server_key: "616e1915dbee0d3609dbda21e4706069"
server_key_iv: "bd689dc330e0949e0dc7ef90d6824a40"
server_key_version: "encryptionkey@0"
I20260926 20:53:14.714278 13602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/master-0/wal/instance:
uuid: "c34ec07649dc4073aacd19b4012322dc"
format_stamp: "Formatted at 2026-09-26 20:53:14 on dist-test-slave-hwlj"
server_key: "616e1915dbee0d3609dbda21e4706069"
server_key_iv: "bd689dc330e0949e0dc7ef90d6824a40"
server_key_version: "encryptionkey@0"
I20260926 20:53:14.715695 13602 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:53:14.716698 13617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:14.716917 13602 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:53:14.717044 13602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "c34ec07649dc4073aacd19b4012322dc"
format_stamp: "Formatted at 2026-09-26 20:53:14 on dist-test-slave-hwlj"
server_key: "616e1915dbee0d3609dbda21e4706069"
server_key_iv: "bd689dc330e0949e0dc7ef90d6824a40"
server_key_version: "encryptionkey@0"
I20260926 20:53:14.717135 13602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:53:14.748723 13602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:53:14.750809 13602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:53:14.750989 13602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:53:14.755215 13602 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40317
I20260926 20:53:14.755306 13669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40317 every 8 connection(s)
I20260926 20:53:14.755667 13602 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:53:14.756791 13670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:53:14.756882 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 13602
I20260926 20:53:14.756983 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/master-0/wal/instance
I20260926 20:53:14.757198 392 external_mini_cluster.cc:1428] Setting key 4b44333ff1c4271c23f1f00bce5a4a43
I20260926 20:53:14.760308 13670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc: Bootstrap starting.
I20260926 20:53:14.761030 13670 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc: Neither blocks nor log segments found. Creating new log.
I20260926 20:53:14.761327 13670 log.cc:824] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc: Log is configured to *not* fsync() on all Append() calls
I20260926 20:53:14.762087 13670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc: No bootstrap required, opened a new log
Sep 26 20:53:14 dist-test-slave-hwlj krb5kdc[13589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:53:14.764760 13670 raft_consensus.cc:348] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c34ec07649dc4073aacd19b4012322dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40317 } }
I20260926 20:53:14.764927 13670 raft_consensus.cc:374] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:53:14.764991 13670 raft_consensus.cc:729] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c34ec07649dc4073aacd19b4012322dc, State: Initialized, Role: FOLLOWER
I20260926 20:53:14.765168 13670 consensus_queue.cc:260] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c34ec07649dc4073aacd19b4012322dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40317 } }
I20260926 20:53:14.765264 13670 raft_consensus.cc:388] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:53:14.765300 13670 raft_consensus.cc:482] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:53:14.765357 13670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:53:14.766201 13670 raft_consensus.cc:504] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c34ec07649dc4073aacd19b4012322dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40317 } }
I20260926 20:53:14.766324 13670 leader_election.cc:302] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c34ec07649dc4073aacd19b4012322dc; no voters:
I20260926 20:53:14.766508 13670 leader_election.cc:288] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:53:14.766584 13675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:53:14.766779 13670 sys_catalog.cc:564] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:53:14.766785 13675 raft_consensus.cc:686] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc [term 1 LEADER]: Becoming Leader. State: Replica: c34ec07649dc4073aacd19b4012322dc, State: Running, Role: LEADER
I20260926 20:53:14.766986 13675 consensus_queue.cc:237] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c34ec07649dc4073aacd19b4012322dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40317 } }
I20260926 20:53:14.767953 13677 sys_catalog.cc:455] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader c34ec07649dc4073aacd19b4012322dc. Latest consensus state: current_term: 1 leader_uuid: "c34ec07649dc4073aacd19b4012322dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c34ec07649dc4073aacd19b4012322dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40317 } } }
I20260926 20:53:14.768033 13677 sys_catalog.cc:458] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc [sys.catalog]: This master's current role is: LEADER
I20260926 20:53:14.768000 13676 sys_catalog.cc:455] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c34ec07649dc4073aacd19b4012322dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c34ec07649dc4073aacd19b4012322dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40317 } } }
I20260926 20:53:14.768080 13676 sys_catalog.cc:458] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc [sys.catalog]: This master's current role is: LEADER
W20260926 20:53:14.768678 13691 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:53:14.768735 13691 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:53:14.768791 13690 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:53:14.768967 13690 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:53:14.769461 13673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:14.757801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41378 (local address 127.0.98.62:40317)
0926 20:53:14.758004 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:14.758013 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:14.758040 (+ 27us) server_negotiation.cc:407] Connection header received
0926 20:53:14.758490 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:14.758498 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:14.758557 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:14.758706 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:14.759529 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:14.760550 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:14.761695 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:14.762195 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:14.762311 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:14.765099 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:14.765102 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:14.765103 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:14.765116 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:14.767107 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:14.767533 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:14.767536 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:14.767537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:14.767640 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:14.768216 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:14.768217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:14.768217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:14.768568 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:14.768690 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:14.769108 (+ 418us) server_negotiation.cc:299] Negotiation successful
0926 20:53:14.769249 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":101,"threads_started":1}
I20260926 20:53:14.772161 13690 catalog_manager.cc:1347] Generated new cluster ID: 4d31bd1307c440b4ad6e21af55ea2962
I20260926 20:53:14.772238 13690 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:53:14.809623 13690 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:53:14.810366 13690 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:53:14.815616 13690 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc: Generated new TSK 0
I20260926 20:53:14.815860 13690 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 20:53:14.847256 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40317
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:53:14.931216 13696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:53:14.931449 13696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:53:14.931475 13696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:53:14.932746 13696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:53:14.932807 13696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:53:14.932837 13696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:53:14.934451 13696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:53:14.934728 13696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:53:14.934999 13696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:53:14.940208 13704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:14.940192 13701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:14.940192 13702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:14.940228 13696 server_base.cc:1034] running on GCE node
I20260926 20:53:14.940675 13696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:53:14.940936 13696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:53:14.942124 13696 hybrid_clock.cc:648] HybridClock initialized: now 1790455994942111 us; error 28 us; skew 500 ppm
Sep 26 20:53:14 dist-test-slave-hwlj krb5kdc[13589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455994, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:53:14.945103 13696 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 20:53:14.945616 13696 webserver.cc:460] Webserver started at http://127.0.98.1:32983/ using document root <none> and password file <none>
I20260926 20:53:14.945874 13696 fs_manager.cc:359] Metadata directory not provided
I20260926 20:53:14.945946 13696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:53:14.946101 13696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:53:14.947202 13696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-0/data/instance:
uuid: "f8d233a0707a4a69abbe2d682788fd00"
format_stamp: "Formatted at 2026-09-26 20:53:14 on dist-test-slave-hwlj"
server_key: "c2a4d2cd252fc5535b31a01d2e2d17c0"
server_key_iv: "894ea1d3b39725f2819d9ccbb7f96d34"
server_key_version: "encryptionkey@0"
I20260926 20:53:14.947578 13696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f8d233a0707a4a69abbe2d682788fd00"
format_stamp: "Formatted at 2026-09-26 20:53:14 on dist-test-slave-hwlj"
server_key: "c2a4d2cd252fc5535b31a01d2e2d17c0"
server_key_iv: "894ea1d3b39725f2819d9ccbb7f96d34"
server_key_version: "encryptionkey@0"
I20260926 20:53:14.949079 13696 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:53:14.950054 13711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:14.950302 13696 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:53:14.950418 13696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-0/wal
uuid: "f8d233a0707a4a69abbe2d682788fd00"
format_stamp: "Formatted at 2026-09-26 20:53:14 on dist-test-slave-hwlj"
server_key: "c2a4d2cd252fc5535b31a01d2e2d17c0"
server_key_iv: "894ea1d3b39725f2819d9ccbb7f96d34"
server_key_version: "encryptionkey@0"
I20260926 20:53:14.950510 13696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:53:14.961086 13696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:53:14.963554 13696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:53:14.963727 13696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:53:14.964002 13696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:53:14.964322 13696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:53:14.964376 13696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:14.964419 13696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:53:14.964447 13696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:14.970865 13696 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36027
I20260926 20:53:14.970916 13824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36027 every 8 connection(s)
I20260926 20:53:14.971279 13696 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:53:14.972283 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 13696
I20260926 20:53:14.972375 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-0/wal/instance
I20260926 20:53:14.972553 392 external_mini_cluster.cc:1428] Setting key e88ef8e70f05ef79711b8a3704073dea
Sep 26 20:53:14 dist-test-slave-hwlj krb5kdc[13589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455994, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 20:53:14.982743 13673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:14.972076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34599 (local address 127.0.98.62:40317)
0926 20:53:14.972738 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:14.972741 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:14.972757 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:53:14.972941 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:14.972942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:14.972969 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:14.973034 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:14.974087 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:14.974845 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:14.976459 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:14.976724 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:14.976798 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:14.979366 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:14.979368 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:14.979368 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:14.979378 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:14.981180 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:14.981677 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:14.981678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:14.981678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:14.981742 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:14.982004 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:14.982005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:14.982006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:14.982143 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:14.982223 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:14.982596 (+ 373us) server_negotiation.cc:299] Negotiation successful
0926 20:53:14.982668 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630}
I20260926 20:53:14.983006 13827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:14.972027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.1:34599)
0926 20:53:14.972232 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:14.972271 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:14.972832 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:14.973094 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:14.973097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:14.973778 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:14.973984 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:14.973989 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:14.974955 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:14.974957 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:14.976352 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:14.976356 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:14.976817 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:14.976818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:14.976977 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:14.977557 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:14.977569 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:14.979271 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:53:14.981269 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:14.981276 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:14.981278 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:14.981508 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:14.981812 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:14.981813 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:14.981813 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:14.981926 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:14.982214 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:53:14.982222 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:53:14.982363 (+ 141us) client_negotiation.cc:769] Sending connection context
0926 20:53:14.982562 (+ 199us) client_negotiation.cc:240] Negotiation successful
0926 20:53:14.982733 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":107,"threads_started":1}
I20260926 20:53:14.983520 13825 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40317
I20260926 20:53:14.983618 13825 heartbeater.cc:464] Registering TS with master...
I20260926 20:53:14.983829 13825 heartbeater.cc:511] Master 127.0.98.62:40317 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 20:53:14.984359 13634 ts_manager.cc:194] Registered new tserver with Master: f8d233a0707a4a69abbe2d682788fd00 (127.0.98.1:36027)
I20260926 20:53:14.985302 13634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34599
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 20:53:14.998934 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40317
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:53:15.079350 13830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:53:15.079582 13830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:53:15.079620 13830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:53:15.080776 13830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:53:15.080852 13830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:53:15.080895 13830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:53:15.082337 13830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:53:15.082612 13830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:53:15.082827 13830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:53:15.088284 13830 server_base.cc:1034] running on GCE node
W20260926 20:53:15.088307 13838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:15.088280 13836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:15.088326 13835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:15.088752 13830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:53:15.088989 13830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:53:15.090140 13830 hybrid_clock.cc:648] HybridClock initialized: now 1790455995090118 us; error 35 us; skew 500 ppm
Sep 26 20:53:15 dist-test-slave-hwlj krb5kdc[13589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455995, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:53:15.092684 13830 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 20:53:15.093109 13830 webserver.cc:460] Webserver started at http://127.0.98.2:42563/ using document root <none> and password file <none>
I20260926 20:53:15.093292 13830 fs_manager.cc:359] Metadata directory not provided
I20260926 20:53:15.093333 13830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:53:15.093448 13830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:53:15.094468 13830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-1/data/instance:
uuid: "86e43b5e22124c6ca7c9ef94ad156f31"
format_stamp: "Formatted at 2026-09-26 20:53:15 on dist-test-slave-hwlj"
server_key: "747bee5fe3fbf14f682d2014cea91db2"
server_key_iv: "7ba6bf0d05e6caf0b49dbf2d7cc1d82b"
server_key_version: "encryptionkey@0"
I20260926 20:53:15.094762 13830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-1/wal/instance:
uuid: "86e43b5e22124c6ca7c9ef94ad156f31"
format_stamp: "Formatted at 2026-09-26 20:53:15 on dist-test-slave-hwlj"
server_key: "747bee5fe3fbf14f682d2014cea91db2"
server_key_iv: "7ba6bf0d05e6caf0b49dbf2d7cc1d82b"
server_key_version: "encryptionkey@0"
I20260926 20:53:15.096040 13830 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:53:15.096724 13845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:15.096884 13830 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:53:15.096962 13830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-1/wal
uuid: "86e43b5e22124c6ca7c9ef94ad156f31"
format_stamp: "Formatted at 2026-09-26 20:53:15 on dist-test-slave-hwlj"
server_key: "747bee5fe3fbf14f682d2014cea91db2"
server_key_iv: "7ba6bf0d05e6caf0b49dbf2d7cc1d82b"
server_key_version: "encryptionkey@0"
I20260926 20:53:15.097014 13830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:53:15.107559 13830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:53:15.109786 13830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:53:15.109905 13830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:53:15.110103 13830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:53:15.110432 13830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:53:15.110467 13830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:15.110491 13830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:53:15.110503 13830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:15.116525 13830 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42061
I20260926 20:53:15.116606 13958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42061 every 8 connection(s)
I20260926 20:53:15.116926 13830 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 20:53:15 dist-test-slave-hwlj krb5kdc[13589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455995, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:53:15.123984 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 13830
I20260926 20:53:15.124099 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-1/wal/instance
I20260926 20:53:15.124320 392 external_mini_cluster.cc:1428] Setting key 5e51c475c9d1db6542070a3ee4833798
I20260926 20:53:15.127368 13673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:15.117699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58353 (local address 127.0.98.62:40317)
0926 20:53:15.117755 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:15.117757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:15.118029 (+ 272us) server_negotiation.cc:407] Connection header received
0926 20:53:15.118594 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:15.118595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:15.118607 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:15.118671 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:15.119640 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.120223 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.121224 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.121424 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.121489 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:15.123913 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:15.123915 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:15.123915 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:15.123922 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:15.125698 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:15.126242 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:15.126243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:15.126243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:15.126297 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:15.126587 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:15.126588 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:15.126588 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:15.126757 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:15.126839 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:15.127138 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 20:53:15.127264 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 20:53:15.127456 13961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:15.117651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.2:58353)
0926 20:53:15.117846 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:15.117889 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:15.118442 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:15.118752 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:15.118755 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:15.119348 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:15.119546 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:15.119550 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.120310 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.120311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:15.121142 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:15.121144 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.121548 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.121549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:15.121670 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:15.122276 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:15.122285 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:15.123812 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:53:15.125811 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:15.125820 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:15.125822 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:15.126162 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:15.126361 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:15.126362 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:15.126362 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:15.126510 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:15.126856 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:53:15.126863 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:53:15.126997 (+ 134us) client_negotiation.cc:769] Sending connection context
0926 20:53:15.127103 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 20:53:15.127246 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":94,"threads_started":1}
I20260926 20:53:15.128444 13959 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40317
I20260926 20:53:15.128585 13959 heartbeater.cc:464] Registering TS with master...
I20260926 20:53:15.128858 13959 heartbeater.cc:511] Master 127.0.98.62:40317 requested a full tablet report, sending...
I20260926 20:53:15.129312 13634 ts_manager.cc:194] Registered new tserver with Master: 86e43b5e22124c6ca7c9ef94ad156f31 (127.0.98.2:42061)
I20260926 20:53:15.129889 13634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58353
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 20:53:15.151363 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40317
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:53:15.233692 13964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:53:15.233911 13964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:53:15.233937 13964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:53:15.235138 13964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:53:15.235194 13964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:53:15.235221 13964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:53:15.236773 13964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:53:15.237042 13964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:53:15.237298 13964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:53:15.242759 13972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:15.242759 13969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:15.242800 13970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:15.242796 13964 server_base.cc:1034] running on GCE node
I20260926 20:53:15.243157 13964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:53:15.243432 13964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:53:15.244611 13964 hybrid_clock.cc:648] HybridClock initialized: now 1790455995244575 us; error 49 us; skew 500 ppm
Sep 26 20:53:15 dist-test-slave-hwlj krb5kdc[13589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455995, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:53:15.247351 13964 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 20:53:15.247963 13964 webserver.cc:460] Webserver started at http://127.0.98.3:43491/ using document root <none> and password file <none>
I20260926 20:53:15.248225 13964 fs_manager.cc:359] Metadata directory not provided
I20260926 20:53:15.248303 13964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:53:15.248464 13964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:53:15.249575 13964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-2/data/instance:
uuid: "501a255b1a0a4f6aba703ba67d771eb2"
format_stamp: "Formatted at 2026-09-26 20:53:15 on dist-test-slave-hwlj"
server_key: "b9a42a4e0e3a57bc620d569a5d9a711e"
server_key_iv: "b0ba9d53c134fad71ac78713a3e8243f"
server_key_version: "encryptionkey@0"
I20260926 20:53:15.249960 13964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-2/wal/instance:
uuid: "501a255b1a0a4f6aba703ba67d771eb2"
format_stamp: "Formatted at 2026-09-26 20:53:15 on dist-test-slave-hwlj"
server_key: "b9a42a4e0e3a57bc620d569a5d9a711e"
server_key_iv: "b0ba9d53c134fad71ac78713a3e8243f"
server_key_version: "encryptionkey@0"
I20260926 20:53:15.251360 13964 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:53:15.252228 13979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:15.252414 13964 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 20:53:15.252512 13964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-2/wal
uuid: "501a255b1a0a4f6aba703ba67d771eb2"
format_stamp: "Formatted at 2026-09-26 20:53:15 on dist-test-slave-hwlj"
server_key: "b9a42a4e0e3a57bc620d569a5d9a711e"
server_key_iv: "b0ba9d53c134fad71ac78713a3e8243f"
server_key_version: "encryptionkey@0"
I20260926 20:53:15.252600 13964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:53:15.270534 13964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:53:15.273033 13964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:53:15.273207 13964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:53:15.273454 13964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:53:15.273777 13964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:53:15.273834 13964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:15.273880 13964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:53:15.273912 13964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:15.281749 13964 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33619
I20260926 20:53:15.281804 14092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33619 every 8 connection(s)
I20260926 20:53:15.282153 13964 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
I20260926 20:53:15.286535 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 13964
I20260926 20:53:15.286669 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-2/wal/instance
I20260926 20:53:15.286865 392 external_mini_cluster.cc:1428] Setting key 938e006424107d9648277cb077b05b34
Sep 26 20:53:15 dist-test-slave-hwlj krb5kdc[13589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455995, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:53:15.292449 13673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:15.282932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47199 (local address 127.0.98.62:40317)
0926 20:53:15.283001 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:15.283004 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:15.283226 (+ 222us) server_negotiation.cc:407] Connection header received
0926 20:53:15.283705 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:15.283706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:15.283718 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:15.283816 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:15.285013 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.285594 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.286621 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.286994 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.287081 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:15.289611 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:15.289613 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:15.289613 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:15.289620 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:15.291062 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:15.291476 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:15.291477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:15.291477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:15.291528 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:15.291866 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:15.291867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:15.291867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:15.291992 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:15.292045 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:15.292323 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 20:53:15.292384 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 20:53:15.292701 14095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:15.282918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.3:47199)
0926 20:53:15.283101 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:15.283139 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:15.283610 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:15.283963 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:15.283967 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:15.284664 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:15.284895 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:15.284900 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.285686 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.285687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:15.286513 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:15.286515 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.287094 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.287095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:15.287278 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:15.287887 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:15.287896 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:15.289521 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:53:15.291151 (+ 1630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:15.291158 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:15.291160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:15.291406 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:15.291615 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:15.291616 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:15.291617 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:15.291738 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:15.292060 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:53:15.292068 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:53:15.292207 (+ 139us) client_negotiation.cc:769] Sending connection context
0926 20:53:15.292298 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 20:53:15.292430 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":98,"threads_started":1}
I20260926 20:53:15.293239 14093 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40317
I20260926 20:53:15.293354 14093 heartbeater.cc:464] Registering TS with master...
I20260926 20:53:15.293588 14093 heartbeater.cc:511] Master 127.0.98.62:40317 requested a full tablet report, sending...
I20260926 20:53:15.294034 13634 ts_manager.cc:194] Registered new tserver with Master: 501a255b1a0a4f6aba703ba67d771eb2 (127.0.98.3:33619)
I20260926 20:53:15.294469 13634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47199
I20260926 20:53:15.298751 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:53:15.306246 13673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:15.300275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41380 (local address 127.0.98.62:40317)
0926 20:53:15.300333 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:15.300336 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:15.300417 (+ 81us) server_negotiation.cc:407] Connection header received
0926 20:53:15.300521 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:15.300522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:15.300534 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:15.300603 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:15.301217 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.301771 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.302460 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.302668 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.302720 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:15.303997 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:15.303998 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:15.303998 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:15.304005 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:15.305207 (+ 1202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:15.305536 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:15.305537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:15.305538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:15.305612 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:15.305782 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:15.305783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:15.305783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:15.305935 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:15.306021 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:15.306104 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 20:53:15.306178 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 20:53:15.307924 13634 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41380:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:53:15.309500 13634 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:53:15.315954 14104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:15.311782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49378 (local address 127.0.98.2:42061)
0926 20:53:15.311954 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:15.311959 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:15.312078 (+ 119us) server_negotiation.cc:407] Connection header received
0926 20:53:15.312170 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:15.312172 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:15.312186 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:15.312260 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:53:15.312512 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.313659 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.315008 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.315482 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.315575 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:15.315639 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:15.315813 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 20:53:15.315866 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":80,"threads_started":1}
I20260926 20:53:15.316001 14105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:15.311819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42516 (local address 127.0.98.1:36027)
0926 20:53:15.312097 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:15.312101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:15.312112 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:53:15.312153 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:15.312156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:15.312171 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:15.312247 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:53:15.312538 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.313396 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.314962 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.315635 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.315700 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:15.315781 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:15.315874 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 20:53:15.315915 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":89,"threads_started":1}
I20260926 20:53:15.316015 14102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:15.311570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33619 (local address 127.0.0.1:38950)
0926 20:53:15.312168 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:15.312177 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:15.312230 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:15.312480 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:15.312481 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:15.312483 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:53:15.312535 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:15.312537 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.313787 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.313787 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:15.314705 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:15.314707 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.315719 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.315720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:15.315826 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:15.315831 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:53:15.315882 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 20:53:15.315930 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"mutex_wait_us":3,"thread_start_us":80,"threads_started":1}
I20260926 20:53:15.316301 14106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:15.311827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38950 (local address 127.0.98.3:33619)
0926 20:53:15.312022 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:15.312026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:15.312242 (+ 216us) server_negotiation.cc:407] Connection header received
0926 20:53:15.312310 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:15.312313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:15.312330 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:15.312402 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:53:15.312630 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.313674 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.314806 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.315547 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.315622 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:15.315682 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:15.316198 (+ 516us) server_negotiation.cc:299] Negotiation successful
0926 20:53:15.316243 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":83,"threads_started":1}
I20260926 20:53:15.316017 14101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:15.311562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36027 (local address 127.0.0.1:42516)
0926 20:53:15.311977 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:15.312005 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:15.312073 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:15.312327 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:15.312330 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:15.312331 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:53:15.312403 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:15.312405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.313509 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.313510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:15.314878 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:15.314880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.315697 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.315698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:15.315795 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:15.315799 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:53:15.315858 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 20:53:15.315926 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":71,"threads_started":1}
I20260926 20:53:15.316308 14103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:15.311562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42061 (local address 127.0.0.1:49378)
0926 20:53:15.311961 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:15.312000 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:15.312090 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:15.312339 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:15.312340 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:15.312342 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:53:15.312429 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:15.312431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.313774 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.313776 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:15.314918 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:15.314920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.315574 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.315575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:15.315666 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:15.315682 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 20:53:15.316195 (+ 513us) client_negotiation.cc:240] Negotiation successful
0926 20:53:15.316237 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":1,"spinlock_wait_cycles":384,"thread_start_us":74,"threads_started":1}
I20260926 20:53:15.317113 13759 tablet_service.cc:1467] Processing CreateTablet for tablet a09a4630538f4c4e854493f9ae0e6882 (DEFAULT_TABLE table=test-table [id=b8468c5907754b7db3caba4d29cb8baf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:53:15.317090 14027 tablet_service.cc:1467] Processing CreateTablet for tablet a09a4630538f4c4e854493f9ae0e6882 (DEFAULT_TABLE table=test-table [id=b8468c5907754b7db3caba4d29cb8baf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:53:15.317236 13893 tablet_service.cc:1467] Processing CreateTablet for tablet a09a4630538f4c4e854493f9ae0e6882 (DEFAULT_TABLE table=test-table [id=b8468c5907754b7db3caba4d29cb8baf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:53:15.317451 13759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a09a4630538f4c4e854493f9ae0e6882. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:53:15.317454 14027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a09a4630538f4c4e854493f9ae0e6882. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:53:15.317514 13893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a09a4630538f4c4e854493f9ae0e6882. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:53:15.320222 14107 tablet_bootstrap.cc:492] T a09a4630538f4c4e854493f9ae0e6882 P f8d233a0707a4a69abbe2d682788fd00: Bootstrap starting.
I20260926 20:53:15.320907 14107 tablet_bootstrap.cc:654] T a09a4630538f4c4e854493f9ae0e6882 P f8d233a0707a4a69abbe2d682788fd00: Neither blocks nor log segments found. Creating new log.
I20260926 20:53:15.320884 14108 tablet_bootstrap.cc:492] T a09a4630538f4c4e854493f9ae0e6882 P 86e43b5e22124c6ca7c9ef94ad156f31: Bootstrap starting.
I20260926 20:53:15.321389 14107 log.cc:824] T a09a4630538f4c4e854493f9ae0e6882 P f8d233a0707a4a69abbe2d682788fd00: Log is configured to *not* fsync() on all Append() calls
I20260926 20:53:15.321411 14108 tablet_bootstrap.cc:654] T a09a4630538f4c4e854493f9ae0e6882 P 86e43b5e22124c6ca7c9ef94ad156f31: Neither blocks nor log segments found. Creating new log.
I20260926 20:53:15.322023 14108 log.cc:824] T a09a4630538f4c4e854493f9ae0e6882 P 86e43b5e22124c6ca7c9ef94ad156f31: Log is configured to *not* fsync() on all Append() calls
I20260926 20:53:15.322289 14107 tablet_bootstrap.cc:492] T a09a4630538f4c4e854493f9ae0e6882 P f8d233a0707a4a69abbe2d682788fd00: No bootstrap required, opened a new log
I20260926 20:53:15.322374 14107 ts_tablet_manager.cc:1397] T a09a4630538f4c4e854493f9ae0e6882 P f8d233a0707a4a69abbe2d682788fd00: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:53:15.322400 14109 tablet_bootstrap.cc:492] T a09a4630538f4c4e854493f9ae0e6882 P 501a255b1a0a4f6aba703ba67d771eb2: Bootstrap starting.
I20260926 20:53:15.322814 14108 tablet_bootstrap.cc:492] T a09a4630538f4c4e854493f9ae0e6882 P 86e43b5e22124c6ca7c9ef94ad156f31: No bootstrap required, opened a new log
I20260926 20:53:15.322888 14108 ts_tablet_manager.cc:1397] T a09a4630538f4c4e854493f9ae0e6882 P 86e43b5e22124c6ca7c9ef94ad156f31: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:53:15.324420 14109 tablet_bootstrap.cc:654] T a09a4630538f4c4e854493f9ae0e6882 P 501a255b1a0a4f6aba703ba67d771eb2: Neither blocks nor log segments found. Creating new log.
I20260926 20:53:15.324591 14108 raft_consensus.cc:348] T a09a4630538f4c4e854493f9ae0e6882 P 86e43b5e22124c6ca7c9ef94ad156f31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d233a0707a4a69abbe2d682788fd00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36027 } } peers { permanent_uuid: "86e43b5e22124c6ca7c9ef94ad156f31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42061 } } peers { permanent_uuid: "501a255b1a0a4f6aba703ba67d771eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33619 } }
I20260926 20:53:15.324791 14108 raft_consensus.cc:374] T a09a4630538f4c4e854493f9ae0e6882 P 86e43b5e22124c6ca7c9ef94ad156f31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:53:15.324687 14107 raft_consensus.cc:348] T a09a4630538f4c4e854493f9ae0e6882 P f8d233a0707a4a69abbe2d682788fd00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d233a0707a4a69abbe2d682788fd00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36027 } } peers { permanent_uuid: "86e43b5e22124c6ca7c9ef94ad156f31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42061 } } peers { permanent_uuid: "501a255b1a0a4f6aba703ba67d771eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33619 } }
I20260926 20:53:15.324859 14107 raft_consensus.cc:374] T a09a4630538f4c4e854493f9ae0e6882 P f8d233a0707a4a69abbe2d682788fd00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:53:15.324888 14108 raft_consensus.cc:729] T a09a4630538f4c4e854493f9ae0e6882 P 86e43b5e22124c6ca7c9ef94ad156f31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86e43b5e22124c6ca7c9ef94ad156f31, State: Initialized, Role: FOLLOWER
I20260926 20:53:15.324921 14107 raft_consensus.cc:729] T a09a4630538f4c4e854493f9ae0e6882 P f8d233a0707a4a69abbe2d682788fd00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8d233a0707a4a69abbe2d682788fd00, State: Initialized, Role: FOLLOWER
I20260926 20:53:15.325029 14109 log.cc:824] T a09a4630538f4c4e854493f9ae0e6882 P 501a255b1a0a4f6aba703ba67d771eb2: Log is configured to *not* fsync() on all Append() calls
I20260926 20:53:15.325039 14107 consensus_queue.cc:260] T a09a4630538f4c4e854493f9ae0e6882 P f8d233a0707a4a69abbe2d682788fd00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d233a0707a4a69abbe2d682788fd00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36027 } } peers { permanent_uuid: "86e43b5e22124c6ca7c9ef94ad156f31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42061 } } peers { permanent_uuid: "501a255b1a0a4f6aba703ba67d771eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33619 } }
I20260926 20:53:15.325052 14108 consensus_queue.cc:260] T a09a4630538f4c4e854493f9ae0e6882 P 86e43b5e22124c6ca7c9ef94ad156f31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d233a0707a4a69abbe2d682788fd00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36027 } } peers { permanent_uuid: "86e43b5e22124c6ca7c9ef94ad156f31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42061 } } peers { permanent_uuid: "501a255b1a0a4f6aba703ba67d771eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33619 } }
I20260926 20:53:15.325338 14107 ts_tablet_manager.cc:1428] T a09a4630538f4c4e854493f9ae0e6882 P f8d233a0707a4a69abbe2d682788fd00: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:53:15.325371 13959 heartbeater.cc:503] Master 127.0.98.62:40317 was elected leader, sending a full tablet report...
I20260926 20:53:15.325481 13825 heartbeater.cc:503] Master 127.0.98.62:40317 was elected leader, sending a full tablet report...
I20260926 20:53:15.325361 14108 ts_tablet_manager.cc:1428] T a09a4630538f4c4e854493f9ae0e6882 P 86e43b5e22124c6ca7c9ef94ad156f31: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 20:53:15.326042 14109 tablet_bootstrap.cc:492] T a09a4630538f4c4e854493f9ae0e6882 P 501a255b1a0a4f6aba703ba67d771eb2: No bootstrap required, opened a new log
I20260926 20:53:15.326118 14109 ts_tablet_manager.cc:1397] T a09a4630538f4c4e854493f9ae0e6882 P 501a255b1a0a4f6aba703ba67d771eb2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 20:53:15.327834 14109 raft_consensus.cc:348] T a09a4630538f4c4e854493f9ae0e6882 P 501a255b1a0a4f6aba703ba67d771eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d233a0707a4a69abbe2d682788fd00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36027 } } peers { permanent_uuid: "86e43b5e22124c6ca7c9ef94ad156f31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42061 } } peers { permanent_uuid: "501a255b1a0a4f6aba703ba67d771eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33619 } }
I20260926 20:53:15.328011 14109 raft_consensus.cc:374] T a09a4630538f4c4e854493f9ae0e6882 P 501a255b1a0a4f6aba703ba67d771eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:53:15.328058 14109 raft_consensus.cc:729] T a09a4630538f4c4e854493f9ae0e6882 P 501a255b1a0a4f6aba703ba67d771eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 501a255b1a0a4f6aba703ba67d771eb2, State: Initialized, Role: FOLLOWER
I20260926 20:53:15.328187 14109 consensus_queue.cc:260] T a09a4630538f4c4e854493f9ae0e6882 P 501a255b1a0a4f6aba703ba67d771eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d233a0707a4a69abbe2d682788fd00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36027 } } peers { permanent_uuid: "86e43b5e22124c6ca7c9ef94ad156f31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42061 } } peers { permanent_uuid: "501a255b1a0a4f6aba703ba67d771eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33619 } }
I20260926 20:53:15.328485 14109 ts_tablet_manager.cc:1428] T a09a4630538f4c4e854493f9ae0e6882 P 501a255b1a0a4f6aba703ba67d771eb2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 20:53:15.328540 14093 heartbeater.cc:503] Master 127.0.98.62:40317 was elected leader, sending a full tablet report...
I20260926 20:53:15.361747 14112 raft_consensus.cc:482] T a09a4630538f4c4e854493f9ae0e6882 P f8d233a0707a4a69abbe2d682788fd00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:53:15.361868 14112 raft_consensus.cc:504] T a09a4630538f4c4e854493f9ae0e6882 P f8d233a0707a4a69abbe2d682788fd00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d233a0707a4a69abbe2d682788fd00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36027 } } peers { permanent_uuid: "86e43b5e22124c6ca7c9ef94ad156f31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42061 } } peers { permanent_uuid: "501a255b1a0a4f6aba703ba67d771eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33619 } }
I20260926 20:53:15.362217 14112 leader_election.cc:288] T a09a4630538f4c4e854493f9ae0e6882 P f8d233a0707a4a69abbe2d682788fd00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86e43b5e22124c6ca7c9ef94ad156f31 (127.0.98.2:42061), 501a255b1a0a4f6aba703ba67d771eb2 (127.0.98.3:33619)
I20260926 20:53:15.366288 14106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:15.362588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46075 (local address 127.0.98.3:33619)
0926 20:53:15.362632 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:15.362634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:15.362778 (+ 144us) server_negotiation.cc:407] Connection header received
0926 20:53:15.362804 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:15.362805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:15.362817 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:15.362873 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:53:15.363021 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.363721 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.365340 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.365928 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.366005 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:15.366015 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:15.366170 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 20:53:15.366209 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 20:53:15.366281 14104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:15.362586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59913 (local address 127.0.98.2:42061)
0926 20:53:15.362632 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:15.362634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:15.362643 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:53:15.362672 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:15.362673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:15.362688 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:15.362786 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:53:15.362959 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.363722 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.365340 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.365928 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.366005 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:15.366014 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:15.366170 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 20:53:15.366209 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 20:53:15.366299 13827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:15.362449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42061 (local address 127.0.98.1:59913)
0926 20:53:15.362499 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:15.362511 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:15.362565 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:15.362789 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:15.362790 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:15.362792 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:53:15.362881 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:15.362883 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.363833 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.363835 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:15.365245 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:15.365248 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.366013 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.366014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:15.366092 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:15.366096 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:53:15.366153 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 20:53:15.366218 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":115}
I20260926 20:53:15.366307 14116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:15.362449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33619 (local address 127.0.98.1:46075)
0926 20:53:15.362669 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:15.362688 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:15.362728 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:15.362882 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:15.362883 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:15.362884 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:53:15.362954 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:15.362955 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.363833 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.363835 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:15.365245 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:15.365248 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.366013 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.366014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:15.366092 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:15.366096 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:53:15.366153 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 20:53:15.366205 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":128,"thread_start_us":105,"threads_started":1}
I20260926 20:53:15.366751 13913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a4630538f4c4e854493f9ae0e6882" candidate_uuid: "f8d233a0707a4a69abbe2d682788fd00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86e43b5e22124c6ca7c9ef94ad156f31" is_pre_election: true
I20260926 20:53:15.366930 13913 raft_consensus.cc:2454] T a09a4630538f4c4e854493f9ae0e6882 P 86e43b5e22124c6ca7c9ef94ad156f31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8d233a0707a4a69abbe2d682788fd00 in term 0.
I20260926 20:53:15.366920 14047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a4630538f4c4e854493f9ae0e6882" candidate_uuid: "f8d233a0707a4a69abbe2d682788fd00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "501a255b1a0a4f6aba703ba67d771eb2" is_pre_election: true
I20260926 20:53:15.367085 14047 raft_consensus.cc:2454] T a09a4630538f4c4e854493f9ae0e6882 P 501a255b1a0a4f6aba703ba67d771eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8d233a0707a4a69abbe2d682788fd00 in term 0.
I20260926 20:53:15.367197 13713 leader_election.cc:302] T a09a4630538f4c4e854493f9ae0e6882 P f8d233a0707a4a69abbe2d682788fd00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86e43b5e22124c6ca7c9ef94ad156f31, f8d233a0707a4a69abbe2d682788fd00; no voters:
I20260926 20:53:15.367364 14112 raft_consensus.cc:2781] T a09a4630538f4c4e854493f9ae0e6882 P f8d233a0707a4a69abbe2d682788fd00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:53:15.367439 14112 raft_consensus.cc:482] T a09a4630538f4c4e854493f9ae0e6882 P f8d233a0707a4a69abbe2d682788fd00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260926 20:53:15.367425 13960 tablet.cc:2369] T a09a4630538f4c4e854493f9ae0e6882 P 86e43b5e22124c6ca7c9ef94ad156f31: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:53:15.367478 14112 raft_consensus.cc:3037] T a09a4630538f4c4e854493f9ae0e6882 P f8d233a0707a4a69abbe2d682788fd00 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:53:15.368269 14112 raft_consensus.cc:504] T a09a4630538f4c4e854493f9ae0e6882 P f8d233a0707a4a69abbe2d682788fd00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d233a0707a4a69abbe2d682788fd00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36027 } } peers { permanent_uuid: "86e43b5e22124c6ca7c9ef94ad156f31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42061 } } peers { permanent_uuid: "501a255b1a0a4f6aba703ba67d771eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33619 } }
I20260926 20:53:15.368420 14112 leader_election.cc:288] T a09a4630538f4c4e854493f9ae0e6882 P f8d233a0707a4a69abbe2d682788fd00 [CANDIDATE]: Term 1 election: Requested vote from peers 86e43b5e22124c6ca7c9ef94ad156f31 (127.0.98.2:42061), 501a255b1a0a4f6aba703ba67d771eb2 (127.0.98.3:33619)
I20260926 20:53:15.368561 13913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a4630538f4c4e854493f9ae0e6882" candidate_uuid: "f8d233a0707a4a69abbe2d682788fd00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86e43b5e22124c6ca7c9ef94ad156f31"
I20260926 20:53:15.368573 14047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a4630538f4c4e854493f9ae0e6882" candidate_uuid: "f8d233a0707a4a69abbe2d682788fd00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "501a255b1a0a4f6aba703ba67d771eb2"
I20260926 20:53:15.368652 13913 raft_consensus.cc:3037] T a09a4630538f4c4e854493f9ae0e6882 P 86e43b5e22124c6ca7c9ef94ad156f31 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:53:15.368651 14047 raft_consensus.cc:3037] T a09a4630538f4c4e854493f9ae0e6882 P 501a255b1a0a4f6aba703ba67d771eb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:53:15.369334 13913 raft_consensus.cc:2454] T a09a4630538f4c4e854493f9ae0e6882 P 86e43b5e22124c6ca7c9ef94ad156f31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8d233a0707a4a69abbe2d682788fd00 in term 1.
I20260926 20:53:15.369385 14047 raft_consensus.cc:2454] T a09a4630538f4c4e854493f9ae0e6882 P 501a255b1a0a4f6aba703ba67d771eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8d233a0707a4a69abbe2d682788fd00 in term 1.
I20260926 20:53:15.369518 13713 leader_election.cc:302] T a09a4630538f4c4e854493f9ae0e6882 P f8d233a0707a4a69abbe2d682788fd00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86e43b5e22124c6ca7c9ef94ad156f31, f8d233a0707a4a69abbe2d682788fd00; no voters:
I20260926 20:53:15.369627 14112 raft_consensus.cc:2781] T a09a4630538f4c4e854493f9ae0e6882 P f8d233a0707a4a69abbe2d682788fd00 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:53:15.369764 14112 raft_consensus.cc:686] T a09a4630538f4c4e854493f9ae0e6882 P f8d233a0707a4a69abbe2d682788fd00 [term 1 LEADER]: Becoming Leader. State: Replica: f8d233a0707a4a69abbe2d682788fd00, State: Running, Role: LEADER
I20260926 20:53:15.369866 14112 consensus_queue.cc:237] T a09a4630538f4c4e854493f9ae0e6882 P f8d233a0707a4a69abbe2d682788fd00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d233a0707a4a69abbe2d682788fd00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36027 } } peers { permanent_uuid: "86e43b5e22124c6ca7c9ef94ad156f31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42061 } } peers { permanent_uuid: "501a255b1a0a4f6aba703ba67d771eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33619 } }
I20260926 20:53:15.370705 13634 catalog_manager.cc:5495] T a09a4630538f4c4e854493f9ae0e6882 P f8d233a0707a4a69abbe2d682788fd00 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8d233a0707a4a69abbe2d682788fd00 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "f8d233a0707a4a69abbe2d682788fd00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d233a0707a4a69abbe2d682788fd00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "86e43b5e22124c6ca7c9ef94ad156f31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "501a255b1a0a4f6aba703ba67d771eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33619 } health_report { overall_health: UNKNOWN } } }
I20260926 20:53:15.376781 14105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:15.374620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42526 (local address 127.0.98.1:36027)
0926 20:53:15.374678 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:15.374679 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:15.374705 (+ 26us) server_negotiation.cc:407] Connection header received
0926 20:53:15.374740 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:15.374740 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:15.374757 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:15.374808 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:53:15.374914 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.375383 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.376167 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.376360 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.376411 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:15.376538 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:53:15.376629 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:53:15.376688 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:15.376712 (+ 24us) server_negotiation.cc:299] Negotiation successful
0926 20:53:15.376729 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 20:53:15.378440 13913 raft_consensus.cc:1261] T a09a4630538f4c4e854493f9ae0e6882 P 86e43b5e22124c6ca7c9ef94ad156f31 [term 1 FOLLOWER]: Refusing update from remote peer f8d233a0707a4a69abbe2d682788fd00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:53:15.378443 14047 raft_consensus.cc:1261] T a09a4630538f4c4e854493f9ae0e6882 P 501a255b1a0a4f6aba703ba67d771eb2 [term 1 FOLLOWER]: Refusing update from remote peer f8d233a0707a4a69abbe2d682788fd00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:53:15.378758 14112 consensus_queue.cc:1035] T a09a4630538f4c4e854493f9ae0e6882 P f8d233a0707a4a69abbe2d682788fd00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "501a255b1a0a4f6aba703ba67d771eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:53:15.378922 14117 consensus_queue.cc:1035] T a09a4630538f4c4e854493f9ae0e6882 P f8d233a0707a4a69abbe2d682788fd00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86e43b5e22124c6ca7c9ef94ad156f31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:53:15.381701 14122 mvcc.cc:205] Tried to move back new op lower bound from 7333707757067485184 to 7333707757035601920. Current Snapshot: MvccSnapshot[applied={T|T < 7333707757067485184}]
I20260926 20:53:15.383249 13634 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:53:15.383349 13634 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:53:15.384078 13634 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc: Sending DeleteTablet for 3 replicas of tablet a09a4630538f4c4e854493f9ae0e6882
I20260926 20:53:15.384392 13759 tablet_service.cc:1514] Processing DeleteTablet for tablet a09a4630538f4c4e854493f9ae0e6882 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:53:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42516
I20260926 20:53:15.384416 13893 tablet_service.cc:1514] Processing DeleteTablet for tablet a09a4630538f4c4e854493f9ae0e6882 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:53:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49378
I20260926 20:53:15.384444 14027 tablet_service.cc:1514] Processing DeleteTablet for tablet a09a4630538f4c4e854493f9ae0e6882 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:53:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38950
I20260926 20:53:15.384603 14131 tablet_replica.cc:330] T a09a4630538f4c4e854493f9ae0e6882 P f8d233a0707a4a69abbe2d682788fd00: stopping tablet replica
I20260926 20:53:15.384693 14131 raft_consensus.cc:2229] T a09a4630538f4c4e854493f9ae0e6882 P f8d233a0707a4a69abbe2d682788fd00 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:53:15.384720 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 13696
I20260926 20:53:15.384814 14131 raft_consensus.cc:2258] T a09a4630538f4c4e854493f9ae0e6882 P f8d233a0707a4a69abbe2d682788fd00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:53:15.385288 14130 tablet_replica.cc:330] T a09a4630538f4c4e854493f9ae0e6882 P 86e43b5e22124c6ca7c9ef94ad156f31: stopping tablet replica
I20260926 20:53:15.385387 14130 raft_consensus.cc:2229] T a09a4630538f4c4e854493f9ae0e6882 P 86e43b5e22124c6ca7c9ef94ad156f31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:53:15.385486 14130 raft_consensus.cc:2258] T a09a4630538f4c4e854493f9ae0e6882 P 86e43b5e22124c6ca7c9ef94ad156f31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:53:15.385883 14130 ts_tablet_manager.cc:1905] T a09a4630538f4c4e854493f9ae0e6882 P 86e43b5e22124c6ca7c9ef94ad156f31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:53:15.387899 14130 ts_tablet_manager.cc:1918] T a09a4630538f4c4e854493f9ae0e6882 P 86e43b5e22124c6ca7c9ef94ad156f31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:53:15.387979 14130 log.cc:1196] T a09a4630538f4c4e854493f9ae0e6882 P 86e43b5e22124c6ca7c9ef94ad156f31: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-1/wal/wals/a09a4630538f4c4e854493f9ae0e6882
I20260926 20:53:15.388247 14130 ts_tablet_manager.cc:1939] T a09a4630538f4c4e854493f9ae0e6882 P 86e43b5e22124c6ca7c9ef94ad156f31: Deleting consensus metadata
I20260926 20:53:15.388810 13619 catalog_manager.cc:4841] TS 86e43b5e22124c6ca7c9ef94ad156f31 (127.0.98.2:42061): tablet a09a4630538f4c4e854493f9ae0e6882 (table test-table [id=b8468c5907754b7db3caba4d29cb8baf]) successfully deleted
W20260926 20:53:15.391443 13620 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36027 (error 108)
I20260926 20:53:15.391641 14132 tablet_replica.cc:330] T a09a4630538f4c4e854493f9ae0e6882 P 501a255b1a0a4f6aba703ba67d771eb2: stopping tablet replica
I20260926 20:53:15.391788 14132 raft_consensus.cc:2229] T a09a4630538f4c4e854493f9ae0e6882 P 501a255b1a0a4f6aba703ba67d771eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:53:15.391909 14132 raft_consensus.cc:2258] T a09a4630538f4c4e854493f9ae0e6882 P 501a255b1a0a4f6aba703ba67d771eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:53:15.391968 14103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:15.391843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36027 (local address 127.0.0.1:42528)
0926 20:53:15.391888 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:15.391936 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260926 20:53:15.392076 13620 catalog_manager.cc:4584] TS f8d233a0707a4a69abbe2d682788fd00 (127.0.98.1:36027): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a09a4630538f4c4e854493f9ae0e6882: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36027: connect: Connection refused (error 111)
I20260926 20:53:15.392122 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 13830
I20260926 20:53:15.396744 14132 ts_tablet_manager.cc:1905] T a09a4630538f4c4e854493f9ae0e6882 P 501a255b1a0a4f6aba703ba67d771eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:53:15.396966 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 13964
I20260926 20:53:15.401351 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 13602
I20260926 20:53:15.401415 14103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:15.401292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33619 (local address 127.0.0.1:38952)
0926 20:53:15.401337 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:15.401367 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.3:33619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260926 20:53:15.407017 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40317
--webserver_interface=127.0.98.62
--webserver_port=42341
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:53:15.488443 14134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:53:15.488649 14134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:53:15.488672 14134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:53:15.489758 14134 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:53:15.489791 14134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:53:15.489806 14134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:53:15.489818 14134 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:53:15.489830 14134 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:53:15.491228 14134 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:53:15.491417 14134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:53:15.491603 14134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:53:15.500233 14140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:15.500339 14134 server_base.cc:1034] running on GCE node
W20260926 20:53:15.500208 14141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:15.500231 14143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:15.500725 14134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:53:15.501133 14134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:53:15.502295 14134 hybrid_clock.cc:648] HybridClock initialized: now 1790455995502282 us; error 43 us; skew 500 ppm
Sep 26 20:53:15 dist-test-slave-hwlj krb5kdc[13589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455995, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:53:15.511328 14134 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:53:15.511965 14134 webserver.cc:460] Webserver started at http://127.0.98.62:42341/ using document root <none> and password file <none>
I20260926 20:53:15.512231 14134 fs_manager.cc:359] Metadata directory not provided
I20260926 20:53:15.512318 14134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:53:15.513794 14134 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:53:15.514472 14150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:15.514672 14134 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:53:15.514809 14134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "c34ec07649dc4073aacd19b4012322dc"
format_stamp: "Formatted at 2026-09-26 20:53:14 on dist-test-slave-hwlj"
server_key: "616e1915dbee0d3609dbda21e4706069"
server_key_iv: "bd689dc330e0949e0dc7ef90d6824a40"
server_key_version: "encryptionkey@0"
I20260926 20:53:15.515240 14134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:53:15.523844 14134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:53:15.526072 14134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:53:15.526266 14134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:53:15.530965 14134 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40317
I20260926 20:53:15.531548 14134 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:53:15.531572 14202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40317 every 8 connection(s)
I20260926 20:53:15.532021 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 14134
I20260926 20:53:15.532333 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:36027
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40317
--webserver_port=32983
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:53:15.533897 14203 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:53:15.534592 14203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc: Bootstrap starting.
I20260926 20:53:15.543109 14203 log.cc:824] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc: Log is configured to *not* fsync() on all Append() calls
I20260926 20:53:15.545218 14203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:53:15.545527 14203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc: Bootstrap complete.
I20260926 20:53:15.547974 14203 raft_consensus.cc:348] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c34ec07649dc4073aacd19b4012322dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40317 } }
I20260926 20:53:15.548240 14203 raft_consensus.cc:729] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c34ec07649dc4073aacd19b4012322dc, State: Initialized, Role: FOLLOWER
I20260926 20:53:15.548429 14203 consensus_queue.cc:260] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c34ec07649dc4073aacd19b4012322dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40317 } }
I20260926 20:53:15.548509 14203 raft_consensus.cc:388] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:53:15.548569 14203 raft_consensus.cc:482] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:53:15.548637 14203 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc [term 1 FOLLOWER]: Advancing to term 2
I20260926 20:53:15.549708 14203 raft_consensus.cc:504] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c34ec07649dc4073aacd19b4012322dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40317 } }
I20260926 20:53:15.549836 14203 leader_election.cc:302] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c34ec07649dc4073aacd19b4012322dc; no voters:
I20260926 20:53:15.550046 14203 leader_election.cc:288] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 20:53:15.550084 14207 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc [term 2 FOLLOWER]: Leader election won for term 2
I20260926 20:53:15.550201 14207 raft_consensus.cc:686] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc [term 2 LEADER]: Becoming Leader. State: Replica: c34ec07649dc4073aacd19b4012322dc, State: Running, Role: LEADER
I20260926 20:53:15.550299 14207 consensus_queue.cc:237] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c34ec07649dc4073aacd19b4012322dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40317 } }
I20260926 20:53:15.550364 14203 sys_catalog.cc:564] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:53:15.550563 14209 sys_catalog.cc:455] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader c34ec07649dc4073aacd19b4012322dc. Latest consensus state: current_term: 2 leader_uuid: "c34ec07649dc4073aacd19b4012322dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c34ec07649dc4073aacd19b4012322dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40317 } } }
I20260926 20:53:15.550664 14209 sys_catalog.cc:458] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc [sys.catalog]: This master's current role is: LEADER
I20260926 20:53:15.550566 14208 sys_catalog.cc:455] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c34ec07649dc4073aacd19b4012322dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c34ec07649dc4073aacd19b4012322dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40317 } } }
I20260926 20:53:15.550840 14208 sys_catalog.cc:458] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc [sys.catalog]: This master's current role is: LEADER
I20260926 20:53:15.550874 14211 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:53:15.551735 14211 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:53:15.551882 14211 catalog_manager.cc:1259] Loaded cluster ID: 4d31bd1307c440b4ad6e21af55ea2962
I20260926 20:53:15.552134 14211 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:53:15.552420 14211 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 20:53:15.555733 14211 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 20:53:15.561197 14211 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:53:15.561441 14211 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc: Loaded TSK: 0
I20260926 20:53:15.561676 14211 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 20:53:15.623885 14205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:53:15.624080 14205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:53:15.624104 14205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:53:15.625262 14205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:53:15.625305 14205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:53:15.625331 14205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:53:15.626775 14205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:36027
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=32983
--tserver_master_addrs=127.0.98.62:40317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:53:15.627035 14205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:53:15.627264 14205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:53:15.633154 14233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:15.633144 14230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:15.633368 14205 server_base.cc:1034] running on GCE node
W20260926 20:53:15.633165 14231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:15.633605 14205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:53:15.633870 14205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:53:15.635030 14205 hybrid_clock.cc:648] HybridClock initialized: now 1790455995635018 us; error 27 us; skew 500 ppm
Sep 26 20:53:15 dist-test-slave-hwlj krb5kdc[13589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455995, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:53:15.637748 14205 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 20:53:15.638284 14205 webserver.cc:460] Webserver started at http://127.0.98.1:32983/ using document root <none> and password file <none>
I20260926 20:53:15.638571 14205 fs_manager.cc:359] Metadata directory not provided
I20260926 20:53:15.638656 14205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:53:15.640054 14205 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:53:15.640761 14240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:15.640954 14205 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:53:15.641052 14205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-0/wal
uuid: "f8d233a0707a4a69abbe2d682788fd00"
format_stamp: "Formatted at 2026-09-26 20:53:14 on dist-test-slave-hwlj"
server_key: "c2a4d2cd252fc5535b31a01d2e2d17c0"
server_key_iv: "894ea1d3b39725f2819d9ccbb7f96d34"
server_key_version: "encryptionkey@0"
I20260926 20:53:15.641409 14205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:53:15.679382 14205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:53:15.681679 14205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:53:15.681867 14205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:53:15.682123 14205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:53:15.682602 14247 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 20:53:15.684455 14205 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 20:53:15.684536 14205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 20:53:15.684589 14205 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 20:53:15.685271 14205 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 20:53:15.685328 14205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 20:53:15.685392 14247 tablet_bootstrap.cc:492] T a09a4630538f4c4e854493f9ae0e6882 P f8d233a0707a4a69abbe2d682788fd00: Bootstrap starting.
I20260926 20:53:15.692431 14205 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36027
I20260926 20:53:15.692533 14354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36027 every 8 connection(s)
I20260926 20:53:15.692869 14205 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:53:15.696451 14247 log.cc:824] T a09a4630538f4c4e854493f9ae0e6882 P f8d233a0707a4a69abbe2d682788fd00: Log is configured to *not* fsync() on all Append() calls
I20260926 20:53:15.697862 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 14205
I20260926 20:53:15.698127 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42061
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40317
--webserver_port=42563
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:53:15.699995 14247 tablet_bootstrap.cc:492] T a09a4630538f4c4e854493f9ae0e6882 P f8d233a0707a4a69abbe2d682788fd00: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:53:15.700341 14247 tablet_bootstrap.cc:492] T a09a4630538f4c4e854493f9ae0e6882 P f8d233a0707a4a69abbe2d682788fd00: Bootstrap complete.
I20260926 20:53:15.701668 14247 ts_tablet_manager.cc:1397] T a09a4630538f4c4e854493f9ae0e6882 P f8d233a0707a4a69abbe2d682788fd00: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.006s
Sep 26 20:53:15 dist-test-slave-hwlj krb5kdc[13589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455995, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:53:15.702844 14247 raft_consensus.cc:348] T a09a4630538f4c4e854493f9ae0e6882 P f8d233a0707a4a69abbe2d682788fd00 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d233a0707a4a69abbe2d682788fd00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36027 } } peers { permanent_uuid: "86e43b5e22124c6ca7c9ef94ad156f31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42061 } } peers { permanent_uuid: "501a255b1a0a4f6aba703ba67d771eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33619 } }
I20260926 20:53:15.703372 14247 raft_consensus.cc:729] T a09a4630538f4c4e854493f9ae0e6882 P f8d233a0707a4a69abbe2d682788fd00 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8d233a0707a4a69abbe2d682788fd00, State: Initialized, Role: FOLLOWER
I20260926 20:53:15.703549 14247 consensus_queue.cc:260] T a09a4630538f4c4e854493f9ae0e6882 P f8d233a0707a4a69abbe2d682788fd00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d233a0707a4a69abbe2d682788fd00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36027 } } peers { permanent_uuid: "86e43b5e22124c6ca7c9ef94ad156f31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42061 } } peers { permanent_uuid: "501a255b1a0a4f6aba703ba67d771eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33619 } }
I20260926 20:53:15.704191 14247 ts_tablet_manager.cc:1428] T a09a4630538f4c4e854493f9ae0e6882 P f8d233a0707a4a69abbe2d682788fd00: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260926 20:53:15.706892 14358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:15.693689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49841 (local address 127.0.98.62:40317)
0926 20:53:15.693866 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:15.693878 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:15.693918 (+ 40us) server_negotiation.cc:407] Connection header received
0926 20:53:15.694821 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:15.694828 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:15.694872 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:15.694996 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:15.696032 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.697325 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.698606 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.699734 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.699892 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:15.702803 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:15.702807 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:15.702807 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:15.702821 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:15.704852 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:15.705417 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:15.705427 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:15.705427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:15.705523 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:15.705897 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:15.705897 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:15.705898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:15.706147 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:15.706224 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:15.706621 (+ 397us) server_negotiation.cc:299] Negotiation successful
0926 20:53:15.706749 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":98,"threads_started":1}
I20260926 20:53:15.707010 14357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:15.693586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.1:49841)
0926 20:53:15.693776 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:15.693821 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:15.694541 (+ 720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:15.695078 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:15.695083 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:15.695530 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:15.695899 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:15.695910 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.697443 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.697444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:15.698496 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:15.698499 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.699873 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.699874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:15.700027 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:15.700695 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:15.700705 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:15.702674 (+ 1969us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:53:15.704941 (+ 2267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:15.704949 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:15.704952 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:15.705302 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:15.705648 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:15.705649 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:15.705649 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:15.705812 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:15.706223 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:53:15.706230 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:53:15.706412 (+ 182us) client_negotiation.cc:769] Sending connection context
0926 20:53:15.706544 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 20:53:15.706760 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":96,"threads_started":1}
I20260926 20:53:15.707888 14355 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40317
I20260926 20:53:15.707996 14355 heartbeater.cc:464] Registering TS with master...
I20260926 20:53:15.708197 14355 heartbeater.cc:511] Master 127.0.98.62:40317 requested a full tablet report, sending...
I20260926 20:53:15.709033 14166 ts_manager.cc:194] Registered new tserver with Master: f8d233a0707a4a69abbe2d682788fd00 (127.0.98.1:36027)
I20260926 20:53:15.710382 14166 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49841
I20260926 20:53:15.711361 14355 heartbeater.cc:503] Master 127.0.98.62:40317 was elected leader, sending a full tablet report...
Sep 26 20:53:15 dist-test-slave-hwlj krb5kdc[13589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455995, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 20:53:15.719271 14363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:15.709765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42534 (local address 127.0.98.1:36027)
0926 20:53:15.710906 (+ 1141us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:15.710911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:15.710920 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:53:15.710952 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:15.710954 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:15.710992 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:15.711054 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:15.711559 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.712832 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.713641 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.714099 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.714168 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:15.716281 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:15.716289 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:15.716289 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:15.716297 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:15.718068 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:15.718472 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:15.718475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:15.718475 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:15.718539 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:15.718716 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:15.718717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:15.718717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:15.718877 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:15.718955 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:15.719091 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 20:53:15.719171 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1111,"thread_start_us":66,"threads_started":1}
I20260926 20:53:15.719383 14362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:15.709720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36027 (local address 127.0.0.1:42534)
0926 20:53:15.710677 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:15.710709 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:15.710782 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:15.711080 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:15.711086 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:15.711427 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:15.711491 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:15.711494 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.712914 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.712915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:15.713560 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:15.713563 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.714173 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.714176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:15.714231 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:15.714733 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:15.714738 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:15.716194 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:53:15.718201 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:15.718207 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:15.718209 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:15.718396 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:15.718603 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:15.718604 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:15.718604 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:15.718659 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:15.718938 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:53:15.718943 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:53:15.718984 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 20:53:15.719046 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 20:53:15.719243 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":930,"thread_start_us":98,"threads_started":1}
I20260926 20:53:15.719854 14289 tablet_service.cc:1514] Processing DeleteTablet for tablet a09a4630538f4c4e854493f9ae0e6882 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:53:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42534
I20260926 20:53:15.720472 14364 tablet_replica.cc:330] stopping tablet replica
I20260926 20:53:15.720594 14364 raft_consensus.cc:2229] T a09a4630538f4c4e854493f9ae0e6882 P f8d233a0707a4a69abbe2d682788fd00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:53:15.720707 14364 raft_consensus.cc:2258] T a09a4630538f4c4e854493f9ae0e6882 P f8d233a0707a4a69abbe2d682788fd00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:53:15.721017 14364 ts_tablet_manager.cc:1905] T a09a4630538f4c4e854493f9ae0e6882 P f8d233a0707a4a69abbe2d682788fd00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:53:15.723269 14364 ts_tablet_manager.cc:1918] T a09a4630538f4c4e854493f9ae0e6882 P f8d233a0707a4a69abbe2d682788fd00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:53:15.723367 14364 log.cc:1196] T a09a4630538f4c4e854493f9ae0e6882 P f8d233a0707a4a69abbe2d682788fd00: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-0/wal/wals/a09a4630538f4c4e854493f9ae0e6882
I20260926 20:53:15.723578 14364 ts_tablet_manager.cc:1939] T a09a4630538f4c4e854493f9ae0e6882 P f8d233a0707a4a69abbe2d682788fd00: Deleting consensus metadata
I20260926 20:53:15.724160 14153 catalog_manager.cc:4841] TS f8d233a0707a4a69abbe2d682788fd00 (127.0.98.1:36027): tablet a09a4630538f4c4e854493f9ae0e6882 (table test-table [id=b8468c5907754b7db3caba4d29cb8baf]) successfully deleted
W20260926 20:53:15.786080 14360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:53:15.786298 14360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:53:15.786322 14360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:53:15.787420 14360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:53:15.787453 14360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:53:15.787479 14360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:53:15.788873 14360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42061
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=42563
--tserver_master_addrs=127.0.98.62:40317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:53:15.789064 14360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:53:15.789237 14360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:53:15.794421 14370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:15.794425 14369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:15.794421 14372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:15.794438 14360 server_base.cc:1034] running on GCE node
I20260926 20:53:15.794812 14360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:53:15.795027 14360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:53:15.796218 14360 hybrid_clock.cc:648] HybridClock initialized: now 1790455995796205 us; error 26 us; skew 500 ppm
Sep 26 20:53:15 dist-test-slave-hwlj krb5kdc[13589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455995, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:53:15.798684 14360 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 20:53:15.799171 14360 webserver.cc:460] Webserver started at http://127.0.98.2:42563/ using document root <none> and password file <none>
I20260926 20:53:15.799363 14360 fs_manager.cc:359] Metadata directory not provided
I20260926 20:53:15.799419 14360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:53:15.800819 14360 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:53:15.801466 14379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:15.801718 14360 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:53:15.801800 14360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-1/wal
uuid: "86e43b5e22124c6ca7c9ef94ad156f31"
format_stamp: "Formatted at 2026-09-26 20:53:15 on dist-test-slave-hwlj"
server_key: "747bee5fe3fbf14f682d2014cea91db2"
server_key_iv: "7ba6bf0d05e6caf0b49dbf2d7cc1d82b"
server_key_version: "encryptionkey@0"
I20260926 20:53:15.802135 14360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:53:15.814321 14360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:53:15.816535 14360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:53:15.816646 14360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:53:15.816833 14360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:53:15.817142 14360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:53:15.817173 14360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:15.817221 14360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:53:15.817257 14360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:15.823153 14360 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42061
I20260926 20:53:15.823202 14492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42061 every 8 connection(s)
I20260926 20:53:15.823544 14360 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
I20260926 20:53:15.824247 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 14360
I20260926 20:53:15.824554 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33619
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40317
--webserver_port=43491
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:53:15 dist-test-slave-hwlj krb5kdc[13589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455995, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:53:15.833328 14358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:15.824330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56617 (local address 127.0.98.62:40317)
0926 20:53:15.824380 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:15.824382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:15.824638 (+ 256us) server_negotiation.cc:407] Connection header received
0926 20:53:15.825165 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:15.825167 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:15.825179 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:15.825238 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:15.826194 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.826703 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.827707 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.827990 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.828074 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:15.830308 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:15.830310 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:15.830310 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:15.830317 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:15.831954 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:15.832360 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:15.832361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:15.832362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:15.832422 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:15.832670 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:15.832671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:15.832671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:15.832787 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:15.832857 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:15.833134 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 20:53:15.833225 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 20:53:15.833518 14495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:15.824335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.2:56617)
0926 20:53:15.824517 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:15.824552 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:15.825063 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:15.825326 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:15.825329 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:15.825902 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:15.826091 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:15.826096 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.826807 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.826808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:15.827607 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:15.827609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.828080 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.828082 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:15.828207 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:15.828826 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:15.828837 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:15.830170 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:53:15.832043 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:15.832050 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:15.832052 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:15.832282 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:15.832487 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:15.832488 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:15.832488 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:15.832600 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:15.832859 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:53:15.832866 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:53:15.833005 (+ 139us) client_negotiation.cc:769] Sending connection context
0926 20:53:15.833113 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 20:53:15.833272 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":92,"threads_started":1}
I20260926 20:53:15.834336 14493 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40317
I20260926 20:53:15.834417 14493 heartbeater.cc:464] Registering TS with master...
I20260926 20:53:15.834604 14493 heartbeater.cc:511] Master 127.0.98.62:40317 requested a full tablet report, sending...
I20260926 20:53:15.834977 14166 ts_manager.cc:194] Registered new tserver with Master: 86e43b5e22124c6ca7c9ef94ad156f31 (127.0.98.2:42061)
I20260926 20:53:15.835489 14166 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56617
W20260926 20:53:15.908454 14496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:53:15.908685 14496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:53:15.908712 14496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:53:15.910004 14496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:53:15.910056 14496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:53:15.910085 14496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:53:15.911573 14496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33619
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43491
--tserver_master_addrs=127.0.98.62:40317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:53:15.911875 14496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:53:15.912097 14496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:53:15.917836 14501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:15.917738 14504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:15.917773 14496 server_base.cc:1034] running on GCE node
W20260926 20:53:15.917732 14502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:15.918217 14496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:53:15.918434 14496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:53:15.919584 14496 hybrid_clock.cc:648] HybridClock initialized: now 1790455995919548 us; error 49 us; skew 500 ppm
Sep 26 20:53:15 dist-test-slave-hwlj krb5kdc[13589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455995, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:53:15.922226 14496 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 20:53:15.922746 14496 webserver.cc:460] Webserver started at http://127.0.98.3:43491/ using document root <none> and password file <none>
I20260926 20:53:15.922998 14496 fs_manager.cc:359] Metadata directory not provided
I20260926 20:53:15.923065 14496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:53:15.924505 14496 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260926 20:53:15.925232 14511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:15.925420 14496 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:53:15.925525 14496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-2/wal
uuid: "501a255b1a0a4f6aba703ba67d771eb2"
format_stamp: "Formatted at 2026-09-26 20:53:15 on dist-test-slave-hwlj"
server_key: "b9a42a4e0e3a57bc620d569a5d9a711e"
server_key_iv: "b0ba9d53c134fad71ac78713a3e8243f"
server_key_version: "encryptionkey@0"
I20260926 20:53:15.925868 14496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:53:15.947340 14496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:53:15.949770 14496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:53:15.949949 14496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:53:15.950193 14496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:53:15.950628 14518 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 20:53:15.952296 14496 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 20:53:15.952363 14496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 20:53:15.952406 14496 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 20:53:15.953070 14496 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 20:53:15.953126 14496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 20:53:15.953148 14518 tablet_bootstrap.cc:492] T a09a4630538f4c4e854493f9ae0e6882 P 501a255b1a0a4f6aba703ba67d771eb2: Bootstrap starting.
I20260926 20:53:15.954716 14518 log.cc:824] T a09a4630538f4c4e854493f9ae0e6882 P 501a255b1a0a4f6aba703ba67d771eb2: Log is configured to *not* fsync() on all Append() calls
I20260926 20:53:15.957347 14518 tablet_bootstrap.cc:492] T a09a4630538f4c4e854493f9ae0e6882 P 501a255b1a0a4f6aba703ba67d771eb2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:53:15.957657 14518 tablet_bootstrap.cc:492] T a09a4630538f4c4e854493f9ae0e6882 P 501a255b1a0a4f6aba703ba67d771eb2: Bootstrap complete.
I20260926 20:53:15.957825 14518 ts_tablet_manager.cc:1397] T a09a4630538f4c4e854493f9ae0e6882 P 501a255b1a0a4f6aba703ba67d771eb2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260926 20:53:15.959101 14518 raft_consensus.cc:348] T a09a4630538f4c4e854493f9ae0e6882 P 501a255b1a0a4f6aba703ba67d771eb2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d233a0707a4a69abbe2d682788fd00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36027 } } peers { permanent_uuid: "86e43b5e22124c6ca7c9ef94ad156f31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42061 } } peers { permanent_uuid: "501a255b1a0a4f6aba703ba67d771eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33619 } }
I20260926 20:53:15.959357 14518 raft_consensus.cc:729] T a09a4630538f4c4e854493f9ae0e6882 P 501a255b1a0a4f6aba703ba67d771eb2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 501a255b1a0a4f6aba703ba67d771eb2, State: Initialized, Role: FOLLOWER
I20260926 20:53:15.959581 14518 consensus_queue.cc:260] T a09a4630538f4c4e854493f9ae0e6882 P 501a255b1a0a4f6aba703ba67d771eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d233a0707a4a69abbe2d682788fd00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36027 } } peers { permanent_uuid: "86e43b5e22124c6ca7c9ef94ad156f31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42061 } } peers { permanent_uuid: "501a255b1a0a4f6aba703ba67d771eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33619 } }
I20260926 20:53:15.959905 14518 ts_tablet_manager.cc:1428] T a09a4630538f4c4e854493f9ae0e6882 P 501a255b1a0a4f6aba703ba67d771eb2: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20260926 20:53:15.960323 14496 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33619
I20260926 20:53:15.960356 14627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33619 every 8 connection(s)
I20260926 20:53:15.960680 14496 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 20:53:15 dist-test-slave-hwlj krb5kdc[13589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455995, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:53:15.969874 14358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:15.961402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38963 (local address 127.0.98.62:40317)
0926 20:53:15.961450 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:15.961452 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:15.961641 (+ 189us) server_negotiation.cc:407] Connection header received
0926 20:53:15.961793 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:15.961794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:15.961803 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:15.961865 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:15.962467 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.962970 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.964098 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.964299 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.964367 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:15.966975 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:15.966976 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:15.966977 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:15.966984 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:15.968450 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:15.968900 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:15.968901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:15.968902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:15.968960 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:15.969211 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:15.969212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:15.969213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:15.969334 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:15.969408 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:15.969686 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 20:53:15.969770 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 20:53:15.970078 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 14496
I20260926 20:53:15.970109 14630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:15.961357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.3:38963)
0926 20:53:15.961512 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:15.961565 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:15.961728 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:15.961927 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:15.961929 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:15.962196 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:15.962387 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:15.962392 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.963095 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.963096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:15.963972 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:15.963975 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.964408 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.964409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:15.964521 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:15.965043 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:15.965051 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:15.966862 (+ 1811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:53:15.968554 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:15.968562 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:15.968564 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:15.968806 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:15.969024 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:15.969025 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:15.969026 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:15.969143 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:15.969400 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:53:15.969406 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:53:15.969553 (+ 147us) client_negotiation.cc:769] Sending connection context
0926 20:53:15.969665 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 20:53:15.969860 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":85,"threads_started":1}
I20260926 20:53:15.970669 14628 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40317
I20260926 20:53:15.970786 14628 heartbeater.cc:464] Registering TS with master...
I20260926 20:53:15.970979 14628 heartbeater.cc:511] Master 127.0.98.62:40317 requested a full tablet report, sending...
I20260926 20:53:15.971566 14166 ts_manager.cc:194] Registered new tserver with Master: 501a255b1a0a4f6aba703ba67d771eb2 (127.0.98.3:33619)
I20260926 20:53:15.972174 14166 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38963
Sep 26 20:53:15 dist-test-slave-hwlj krb5kdc[13589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455995, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260926 20:53:15.977427 14358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:15.970527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41382 (local address 127.0.98.62:40317)
0926 20:53:15.970584 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:15.970586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:15.970727 (+ 141us) server_negotiation.cc:407] Connection header received
0926 20:53:15.970811 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:15.970812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:15.970823 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:15.970882 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:15.971495 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.972286 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.973093 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.973377 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.973451 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:15.974638 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:15.974640 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:15.974640 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:15.974647 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:15.976278 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:15.976628 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:15.976629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:15.976630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:15.976685 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:15.976908 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:15.976909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:15.976909 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:15.977020 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:15.977200 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:15.977236 (+ 36us) server_negotiation.cc:299] Negotiation successful
0926 20:53:15.977321 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 20:53:15.978188 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:53:15.979571 14362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:15.971909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33619 (local address 127.0.0.1:38958)
0926 20:53:15.971959 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:15.971969 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:15.972006 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:15.972270 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:15.972271 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:15.972432 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:15.972473 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:15.972475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.973537 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.973538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:15.974208 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:15.974210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.974728 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.974731 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:15.974783 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:15.975183 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:15.975187 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:15.976447 (+ 1260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:53:15.978487 (+ 2040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:15.978488 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:15.978489 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:15.978688 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:15.978916 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:15.978917 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:15.978917 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:15.978974 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:15.979267 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:53:15.979268 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:53:15.979311 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 20:53:15.979376 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 20:53:15.979475 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 20:53:15.979586 14632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:15.971962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38958 (local address 127.0.98.3:33619)
0926 20:53:15.972118 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:15.972122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:15.972133 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:53:15.972166 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:15.972167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:15.972178 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:15.972256 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:15.972553 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.973430 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.974307 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.974654 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.974729 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:15.976535 (+ 1806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:15.976544 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:15.976544 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:15.976551 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:15.978403 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:15.978787 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:15.978791 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:15.978791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:15.978853 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:15.979045 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:15.979046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:15.979046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:15.979192 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:15.979288 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:15.979404 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 20:53:15.979487 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":98,"threads_started":1}
I20260926 20:53:15.980022 14561 tablet_service.cc:1514] Processing DeleteTablet for tablet a09a4630538f4c4e854493f9ae0e6882 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:53:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38958
I20260926 20:53:15.980679 14634 tablet_replica.cc:330] stopping tablet replica
I20260926 20:53:15.980780 14634 raft_consensus.cc:2229] T a09a4630538f4c4e854493f9ae0e6882 P 501a255b1a0a4f6aba703ba67d771eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:53:15.980834 14634 raft_consensus.cc:2258] T a09a4630538f4c4e854493f9ae0e6882 P 501a255b1a0a4f6aba703ba67d771eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:53:15.981142 14634 ts_tablet_manager.cc:1905] T a09a4630538f4c4e854493f9ae0e6882 P 501a255b1a0a4f6aba703ba67d771eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:53:15.981479 14358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:15.978556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41386 (local address 127.0.98.62:40317)
0926 20:53:15.978596 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:15.978598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:15.978709 (+ 111us) server_negotiation.cc:407] Connection header received
0926 20:53:15.978787 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:15.978788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:15.978823 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:15.978908 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:53:15.979002 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.979460 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.980579 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.980884 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.980962 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:15.981154 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:53:15.981271 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:53:15.981377 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:15.981395 (+ 18us) server_negotiation.cc:299] Negotiation successful
0926 20:53:15.981411 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 20:53:15.983444 14634 ts_tablet_manager.cc:1918] T a09a4630538f4c4e854493f9ae0e6882 P 501a255b1a0a4f6aba703ba67d771eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:53:15.983521 14634 log.cc:1196] T a09a4630538f4c4e854493f9ae0e6882 P 501a255b1a0a4f6aba703ba67d771eb2: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-2/wal/wals/a09a4630538f4c4e854493f9ae0e6882
I20260926 20:53:15.983801 14634 ts_tablet_manager.cc:1939] T a09a4630538f4c4e854493f9ae0e6882 P 501a255b1a0a4f6aba703ba67d771eb2: Deleting consensus metadata
I20260926 20:53:15.981727 14166 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:41386:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:53:15.984193 14166 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:53:15.984338 14151 catalog_manager.cc:4841] TS 501a255b1a0a4f6aba703ba67d771eb2 (127.0.98.3:33619): tablet a09a4630538f4c4e854493f9ae0e6882 (table test-table [id=b8468c5907754b7db3caba4d29cb8baf]) successfully deleted
I20260926 20:53:15.986133 14561 tablet_service.cc:1467] Processing CreateTablet for tablet b708fc2e4c664be3b876c6ed919b6079 (DEFAULT_TABLE table=test-table [id=a193f80a26f94e6a88891dd007c7adc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:53:15.986155 14289 tablet_service.cc:1467] Processing CreateTablet for tablet b708fc2e4c664be3b876c6ed919b6079 (DEFAULT_TABLE table=test-table [id=a193f80a26f94e6a88891dd007c7adc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:53:15.986311 14561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b708fc2e4c664be3b876c6ed919b6079. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:53:15.986344 14289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b708fc2e4c664be3b876c6ed919b6079. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:53:15.987542 14518 tablet_bootstrap.cc:492] T b708fc2e4c664be3b876c6ed919b6079 P 501a255b1a0a4f6aba703ba67d771eb2: Bootstrap starting.
I20260926 20:53:15.987589 14247 tablet_bootstrap.cc:492] T b708fc2e4c664be3b876c6ed919b6079 P f8d233a0707a4a69abbe2d682788fd00: Bootstrap starting.
I20260926 20:53:15.987975 14518 tablet_bootstrap.cc:654] T b708fc2e4c664be3b876c6ed919b6079 P 501a255b1a0a4f6aba703ba67d771eb2: Neither blocks nor log segments found. Creating new log.
I20260926 20:53:15.987967 14247 tablet_bootstrap.cc:654] T b708fc2e4c664be3b876c6ed919b6079 P f8d233a0707a4a69abbe2d682788fd00: Neither blocks nor log segments found. Creating new log.
I20260926 20:53:15.988538 14518 tablet_bootstrap.cc:492] T b708fc2e4c664be3b876c6ed919b6079 P 501a255b1a0a4f6aba703ba67d771eb2: No bootstrap required, opened a new log
I20260926 20:53:15.988590 14518 ts_tablet_manager.cc:1397] T b708fc2e4c664be3b876c6ed919b6079 P 501a255b1a0a4f6aba703ba67d771eb2: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 20:53:15.988734 14518 raft_consensus.cc:348] T b708fc2e4c664be3b876c6ed919b6079 P 501a255b1a0a4f6aba703ba67d771eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86e43b5e22124c6ca7c9ef94ad156f31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42061 } } peers { permanent_uuid: "501a255b1a0a4f6aba703ba67d771eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33619 } } peers { permanent_uuid: "f8d233a0707a4a69abbe2d682788fd00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36027 } }
I20260926 20:53:15.988807 14518 raft_consensus.cc:374] T b708fc2e4c664be3b876c6ed919b6079 P 501a255b1a0a4f6aba703ba67d771eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:53:15.988844 14518 raft_consensus.cc:729] T b708fc2e4c664be3b876c6ed919b6079 P 501a255b1a0a4f6aba703ba67d771eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 501a255b1a0a4f6aba703ba67d771eb2, State: Initialized, Role: FOLLOWER
I20260926 20:53:15.988895 14518 consensus_queue.cc:260] T b708fc2e4c664be3b876c6ed919b6079 P 501a255b1a0a4f6aba703ba67d771eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86e43b5e22124c6ca7c9ef94ad156f31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42061 } } peers { permanent_uuid: "501a255b1a0a4f6aba703ba67d771eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33619 } } peers { permanent_uuid: "f8d233a0707a4a69abbe2d682788fd00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36027 } }
I20260926 20:53:15.989039 14628 heartbeater.cc:503] Master 127.0.98.62:40317 was elected leader, sending a full tablet report...
I20260926 20:53:15.989174 14518 ts_tablet_manager.cc:1428] T b708fc2e4c664be3b876c6ed919b6079 P 501a255b1a0a4f6aba703ba67d771eb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 20:53:15.989418 14247 tablet_bootstrap.cc:492] T b708fc2e4c664be3b876c6ed919b6079 P f8d233a0707a4a69abbe2d682788fd00: No bootstrap required, opened a new log
I20260926 20:53:15.989470 14247 ts_tablet_manager.cc:1397] T b708fc2e4c664be3b876c6ed919b6079 P f8d233a0707a4a69abbe2d682788fd00: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:53:15.989796 14247 raft_consensus.cc:348] T b708fc2e4c664be3b876c6ed919b6079 P f8d233a0707a4a69abbe2d682788fd00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86e43b5e22124c6ca7c9ef94ad156f31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42061 } } peers { permanent_uuid: "501a255b1a0a4f6aba703ba67d771eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33619 } } peers { permanent_uuid: "f8d233a0707a4a69abbe2d682788fd00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36027 } }
I20260926 20:53:15.989874 14247 raft_consensus.cc:374] T b708fc2e4c664be3b876c6ed919b6079 P f8d233a0707a4a69abbe2d682788fd00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:53:15.989929 14247 raft_consensus.cc:729] T b708fc2e4c664be3b876c6ed919b6079 P f8d233a0707a4a69abbe2d682788fd00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8d233a0707a4a69abbe2d682788fd00, State: Initialized, Role: FOLLOWER
I20260926 20:53:15.989972 14247 consensus_queue.cc:260] T b708fc2e4c664be3b876c6ed919b6079 P f8d233a0707a4a69abbe2d682788fd00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86e43b5e22124c6ca7c9ef94ad156f31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42061 } } peers { permanent_uuid: "501a255b1a0a4f6aba703ba67d771eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33619 } } peers { permanent_uuid: "f8d233a0707a4a69abbe2d682788fd00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36027 } }
I20260926 20:53:15.990056 14247 ts_tablet_manager.cc:1428] T b708fc2e4c664be3b876c6ed919b6079 P f8d233a0707a4a69abbe2d682788fd00: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 20:53:15.991655 14362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:15.985970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42061 (local address 127.0.0.1:49392)
0926 20:53:15.987561 (+ 1591us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:15.987575 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:15.987628 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:15.987900 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:15.987901 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:15.987903 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:53:15.987951 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:15.987953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.989327 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.989328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:15.990543 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:15.990562 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.991368 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.991369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:15.991439 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:15.991461 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 20:53:15.991522 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 20:53:15.991580 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1569}
I20260926 20:53:15.991659 14635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:15.986119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49392 (local address 127.0.98.2:42061)
0926 20:53:15.986342 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:15.986347 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:15.987689 (+ 1342us) server_negotiation.cc:407] Connection header received
0926 20:53:15.987774 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:15.987777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:15.987795 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:15.988029 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:53:15.988111 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.989201 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.990682 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.991251 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.991368 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:15.991433 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:15.991543 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 20:53:15.991584 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":103,"threads_started":1}
I20260926 20:53:15.992434 14427 tablet_service.cc:1467] Processing CreateTablet for tablet b708fc2e4c664be3b876c6ed919b6079 (DEFAULT_TABLE table=test-table [id=a193f80a26f94e6a88891dd007c7adc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:53:15.992717 14427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b708fc2e4c664be3b876c6ed919b6079. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:53:15.995419 14638 tablet_bootstrap.cc:492] T b708fc2e4c664be3b876c6ed919b6079 P 86e43b5e22124c6ca7c9ef94ad156f31: Bootstrap starting.
I20260926 20:53:15.996064 14638 tablet_bootstrap.cc:654] T b708fc2e4c664be3b876c6ed919b6079 P 86e43b5e22124c6ca7c9ef94ad156f31: Neither blocks nor log segments found. Creating new log.
I20260926 20:53:15.996532 14638 log.cc:824] T b708fc2e4c664be3b876c6ed919b6079 P 86e43b5e22124c6ca7c9ef94ad156f31: Log is configured to *not* fsync() on all Append() calls
I20260926 20:53:15.997221 14638 tablet_bootstrap.cc:492] T b708fc2e4c664be3b876c6ed919b6079 P 86e43b5e22124c6ca7c9ef94ad156f31: No bootstrap required, opened a new log
I20260926 20:53:15.997292 14638 ts_tablet_manager.cc:1397] T b708fc2e4c664be3b876c6ed919b6079 P 86e43b5e22124c6ca7c9ef94ad156f31: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 20:53:15.998709 14638 raft_consensus.cc:348] T b708fc2e4c664be3b876c6ed919b6079 P 86e43b5e22124c6ca7c9ef94ad156f31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86e43b5e22124c6ca7c9ef94ad156f31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42061 } } peers { permanent_uuid: "501a255b1a0a4f6aba703ba67d771eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33619 } } peers { permanent_uuid: "f8d233a0707a4a69abbe2d682788fd00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36027 } }
I20260926 20:53:15.998852 14638 raft_consensus.cc:374] T b708fc2e4c664be3b876c6ed919b6079 P 86e43b5e22124c6ca7c9ef94ad156f31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:53:15.998874 14638 raft_consensus.cc:729] T b708fc2e4c664be3b876c6ed919b6079 P 86e43b5e22124c6ca7c9ef94ad156f31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86e43b5e22124c6ca7c9ef94ad156f31, State: Initialized, Role: FOLLOWER
I20260926 20:53:15.999034 14638 consensus_queue.cc:260] T b708fc2e4c664be3b876c6ed919b6079 P 86e43b5e22124c6ca7c9ef94ad156f31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86e43b5e22124c6ca7c9ef94ad156f31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42061 } } peers { permanent_uuid: "501a255b1a0a4f6aba703ba67d771eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33619 } } peers { permanent_uuid: "f8d233a0707a4a69abbe2d682788fd00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36027 } }
I20260926 20:53:15.999380 14638 ts_tablet_manager.cc:1428] T b708fc2e4c664be3b876c6ed919b6079 P 86e43b5e22124c6ca7c9ef94ad156f31: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 20:53:15.999384 14493 heartbeater.cc:503] Master 127.0.98.62:40317 was elected leader, sending a full tablet report...
I20260926 20:53:16.004279 14640 raft_consensus.cc:482] T b708fc2e4c664be3b876c6ed919b6079 P 86e43b5e22124c6ca7c9ef94ad156f31 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:53:16.004355 14640 raft_consensus.cc:504] T b708fc2e4c664be3b876c6ed919b6079 P 86e43b5e22124c6ca7c9ef94ad156f31 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86e43b5e22124c6ca7c9ef94ad156f31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42061 } } peers { permanent_uuid: "501a255b1a0a4f6aba703ba67d771eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33619 } } peers { permanent_uuid: "f8d233a0707a4a69abbe2d682788fd00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36027 } }
I20260926 20:53:16.004704 14640 leader_election.cc:288] T b708fc2e4c664be3b876c6ed919b6079 P 86e43b5e22124c6ca7c9ef94ad156f31 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 501a255b1a0a4f6aba703ba67d771eb2 (127.0.98.3:33619), f8d233a0707a4a69abbe2d682788fd00 (127.0.98.1:36027)
I20260926 20:53:16.007838 14632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:16.004981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35841 (local address 127.0.98.3:33619)
0926 20:53:16.005028 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:16.005031 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:16.005142 (+ 111us) server_negotiation.cc:407] Connection header received
0926 20:53:16.005168 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:16.005168 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:16.005190 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:16.005254 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:53:16.005449 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:16.006004 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:16.006900 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:16.007472 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:16.007550 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:16.007628 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:16.007686 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 20:53:16.007720 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 20:53:16.007853 14641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:16.004882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33619 (local address 127.0.98.2:35841)
0926 20:53:16.005051 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:16.005059 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:16.005105 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:16.005271 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:16.005271 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:16.005273 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:53:16.005333 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:16.005334 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:16.006095 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:16.006096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:16.006819 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:16.006821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:16.007548 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:16.007549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:16.007615 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:16.007619 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:53:16.007680 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 20:53:16.007719 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":20,"thread_start_us":82,"threads_started":1}
I20260926 20:53:16.007854 14363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:16.004999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49375 (local address 127.0.98.1:36027)
0926 20:53:16.005041 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:16.005043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:16.005051 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:53:16.005076 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:16.005077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:16.005107 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:16.005176 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:53:16.005384 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:16.005868 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:16.006856 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:16.007481 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:16.007571 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:16.007644 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:16.007717 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 20:53:16.007784 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 20:53:16.007856 14495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:16.004882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36027 (local address 127.0.98.2:49375)
0926 20:53:16.004940 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:16.004952 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:16.004991 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:16.005186 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:16.005187 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:16.005189 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:53:16.005284 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:16.005286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:16.005987 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:16.005988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:16.006757 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:16.006759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:16.007572 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:16.007573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:16.007649 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:16.007653 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:53:16.007710 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 20:53:16.007780 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 20:53:16.008129 14581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b708fc2e4c664be3b876c6ed919b6079" candidate_uuid: "86e43b5e22124c6ca7c9ef94ad156f31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "501a255b1a0a4f6aba703ba67d771eb2" is_pre_election: true
I20260926 20:53:16.008273 14581 raft_consensus.cc:2454] T b708fc2e4c664be3b876c6ed919b6079 P 501a255b1a0a4f6aba703ba67d771eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86e43b5e22124c6ca7c9ef94ad156f31 in term 0.
I20260926 20:53:16.008358 14309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b708fc2e4c664be3b876c6ed919b6079" candidate_uuid: "86e43b5e22124c6ca7c9ef94ad156f31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8d233a0707a4a69abbe2d682788fd00" is_pre_election: true
I20260926 20:53:16.008477 14380 leader_election.cc:302] T b708fc2e4c664be3b876c6ed919b6079 P 86e43b5e22124c6ca7c9ef94ad156f31 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 501a255b1a0a4f6aba703ba67d771eb2, 86e43b5e22124c6ca7c9ef94ad156f31; no voters:
I20260926 20:53:16.008531 14309 raft_consensus.cc:2454] T b708fc2e4c664be3b876c6ed919b6079 P f8d233a0707a4a69abbe2d682788fd00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86e43b5e22124c6ca7c9ef94ad156f31 in term 0.
I20260926 20:53:16.008602 14640 raft_consensus.cc:2781] T b708fc2e4c664be3b876c6ed919b6079 P 86e43b5e22124c6ca7c9ef94ad156f31 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:53:16.008654 14640 raft_consensus.cc:482] T b708fc2e4c664be3b876c6ed919b6079 P 86e43b5e22124c6ca7c9ef94ad156f31 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:53:16.008684 14640 raft_consensus.cc:3037] T b708fc2e4c664be3b876c6ed919b6079 P 86e43b5e22124c6ca7c9ef94ad156f31 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:53:16.009328 14640 raft_consensus.cc:504] T b708fc2e4c664be3b876c6ed919b6079 P 86e43b5e22124c6ca7c9ef94ad156f31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86e43b5e22124c6ca7c9ef94ad156f31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42061 } } peers { permanent_uuid: "501a255b1a0a4f6aba703ba67d771eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33619 } } peers { permanent_uuid: "f8d233a0707a4a69abbe2d682788fd00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36027 } }
I20260926 20:53:16.009457 14640 leader_election.cc:288] T b708fc2e4c664be3b876c6ed919b6079 P 86e43b5e22124c6ca7c9ef94ad156f31 [CANDIDATE]: Term 1 election: Requested vote from peers 501a255b1a0a4f6aba703ba67d771eb2 (127.0.98.3:33619), f8d233a0707a4a69abbe2d682788fd00 (127.0.98.1:36027)
I20260926 20:53:16.009601 14581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b708fc2e4c664be3b876c6ed919b6079" candidate_uuid: "86e43b5e22124c6ca7c9ef94ad156f31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "501a255b1a0a4f6aba703ba67d771eb2"
I20260926 20:53:16.009652 14309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b708fc2e4c664be3b876c6ed919b6079" candidate_uuid: "86e43b5e22124c6ca7c9ef94ad156f31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8d233a0707a4a69abbe2d682788fd00"
I20260926 20:53:16.009716 14309 raft_consensus.cc:3037] T b708fc2e4c664be3b876c6ed919b6079 P f8d233a0707a4a69abbe2d682788fd00 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:53:16.009706 14581 raft_consensus.cc:3037] T b708fc2e4c664be3b876c6ed919b6079 P 501a255b1a0a4f6aba703ba67d771eb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:53:16.010519 14309 raft_consensus.cc:2454] T b708fc2e4c664be3b876c6ed919b6079 P f8d233a0707a4a69abbe2d682788fd00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86e43b5e22124c6ca7c9ef94ad156f31 in term 1.
I20260926 20:53:16.010519 14581 raft_consensus.cc:2454] T b708fc2e4c664be3b876c6ed919b6079 P 501a255b1a0a4f6aba703ba67d771eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86e43b5e22124c6ca7c9ef94ad156f31 in term 1.
I20260926 20:53:16.010699 14380 leader_election.cc:302] T b708fc2e4c664be3b876c6ed919b6079 P 86e43b5e22124c6ca7c9ef94ad156f31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 501a255b1a0a4f6aba703ba67d771eb2, 86e43b5e22124c6ca7c9ef94ad156f31; no voters:
I20260926 20:53:16.010808 14640 raft_consensus.cc:2781] T b708fc2e4c664be3b876c6ed919b6079 P 86e43b5e22124c6ca7c9ef94ad156f31 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:53:16.010972 14640 raft_consensus.cc:686] T b708fc2e4c664be3b876c6ed919b6079 P 86e43b5e22124c6ca7c9ef94ad156f31 [term 1 LEADER]: Becoming Leader. State: Replica: 86e43b5e22124c6ca7c9ef94ad156f31, State: Running, Role: LEADER
I20260926 20:53:16.011075 14640 consensus_queue.cc:237] T b708fc2e4c664be3b876c6ed919b6079 P 86e43b5e22124c6ca7c9ef94ad156f31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86e43b5e22124c6ca7c9ef94ad156f31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42061 } } peers { permanent_uuid: "501a255b1a0a4f6aba703ba67d771eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33619 } } peers { permanent_uuid: "f8d233a0707a4a69abbe2d682788fd00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36027 } }
I20260926 20:53:16.011894 14166 catalog_manager.cc:5495] T b708fc2e4c664be3b876c6ed919b6079 P 86e43b5e22124c6ca7c9ef94ad156f31 reported cstate change: term changed from 0 to 1, leader changed from <none> to 86e43b5e22124c6ca7c9ef94ad156f31 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "86e43b5e22124c6ca7c9ef94ad156f31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86e43b5e22124c6ca7c9ef94ad156f31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "501a255b1a0a4f6aba703ba67d771eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8d233a0707a4a69abbe2d682788fd00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36027 } health_report { overall_health: UNKNOWN } } }
I20260926 20:53:16.017649 14635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:16.015358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49406 (local address 127.0.98.2:42061)
0926 20:53:16.015400 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:16.015402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:16.015412 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:53:16.015479 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:16.015479 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:16.015498 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:16.015588 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:53:16.015686 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:16.016184 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:16.016913 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:16.017116 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:16.017189 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:16.017348 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:53:16.017446 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:53:16.017535 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:16.017554 (+ 19us) server_negotiation.cc:299] Negotiation successful
0926 20:53:16.017573 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 20:53:16.019255 14581 raft_consensus.cc:1261] T b708fc2e4c664be3b876c6ed919b6079 P 501a255b1a0a4f6aba703ba67d771eb2 [term 1 FOLLOWER]: Refusing update from remote peer 86e43b5e22124c6ca7c9ef94ad156f31: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:53:16.019258 14309 raft_consensus.cc:1261] T b708fc2e4c664be3b876c6ed919b6079 P f8d233a0707a4a69abbe2d682788fd00 [term 1 FOLLOWER]: Refusing update from remote peer 86e43b5e22124c6ca7c9ef94ad156f31: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:53:16.019541 14642 consensus_queue.cc:1035] T b708fc2e4c664be3b876c6ed919b6079 P 86e43b5e22124c6ca7c9ef94ad156f31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "501a255b1a0a4f6aba703ba67d771eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:53:16.019662 14640 consensus_queue.cc:1035] T b708fc2e4c664be3b876c6ed919b6079 P 86e43b5e22124c6ca7c9ef94ad156f31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8d233a0707a4a69abbe2d682788fd00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:53:16.023854 14167 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:41386:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:53:16.023954 14167 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:41386:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:53:16.024665 14167 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc: Sending DeleteTablet for 3 replicas of tablet b708fc2e4c664be3b876c6ed919b6079
I20260926 20:53:16.024996 14561 tablet_service.cc:1514] Processing DeleteTablet for tablet b708fc2e4c664be3b876c6ed919b6079 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:53:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38958
I20260926 20:53:16.025022 14427 tablet_service.cc:1514] Processing DeleteTablet for tablet b708fc2e4c664be3b876c6ed919b6079 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:53:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49392
I20260926 20:53:16.025187 14634 tablet_replica.cc:330] T b708fc2e4c664be3b876c6ed919b6079 P 501a255b1a0a4f6aba703ba67d771eb2: stopping tablet replica
I20260926 20:53:16.025199 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 14205
I20260926 20:53:16.025267 14634 raft_consensus.cc:2229] T b708fc2e4c664be3b876c6ed919b6079 P 501a255b1a0a4f6aba703ba67d771eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:53:16.025323 14634 raft_consensus.cc:2258] T b708fc2e4c664be3b876c6ed919b6079 P 501a255b1a0a4f6aba703ba67d771eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:53:16.028021 14634 ts_tablet_manager.cc:1905] T b708fc2e4c664be3b876c6ed919b6079 P 501a255b1a0a4f6aba703ba67d771eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:53:16.029943 14634 ts_tablet_manager.cc:1918] T b708fc2e4c664be3b876c6ed919b6079 P 501a255b1a0a4f6aba703ba67d771eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:53:16.030009 14634 log.cc:1196] T b708fc2e4c664be3b876c6ed919b6079 P 501a255b1a0a4f6aba703ba67d771eb2: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-2/wal/wals/b708fc2e4c664be3b876c6ed919b6079
I20260926 20:53:16.030217 14634 ts_tablet_manager.cc:1939] T b708fc2e4c664be3b876c6ed919b6079 P 501a255b1a0a4f6aba703ba67d771eb2: Deleting consensus metadata
I20260926 20:53:16.030925 14151 catalog_manager.cc:4841] TS 501a255b1a0a4f6aba703ba67d771eb2 (127.0.98.3:33619): tablet b708fc2e4c664be3b876c6ed919b6079 (table test-table [id=a193f80a26f94e6a88891dd007c7adc7]) successfully deleted
W20260926 20:53:16.031162 14153 connection.cc:535] client connection to 127.0.98.1:36027 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 20:53:16.031265 14153 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 20:53:16.031558 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 14360
I20260926 20:53:16.031702 14362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:16.031591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36027 (local address 127.0.0.1:42544)
0926 20:53:16.031639 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:16.031674 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 20:53:16.032308 14153 catalog_manager.cc:4584] TS f8d233a0707a4a69abbe2d682788fd00 (127.0.98.1:36027): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b708fc2e4c664be3b876c6ed919b6079: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36027: connect: Connection refused (error 111)
I20260926 20:53:16.037276 14362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:16.037148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42061 (local address 127.0.0.1:49420)
0926 20:53:16.037211 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:16.037245 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:42061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 20:53:16.037300 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 14496
I20260926 20:53:16.041800 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 14134
I20260926 20:53:16.047035 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40317
--webserver_interface=127.0.98.62
--webserver_port=42341
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40317
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:53:16.128180 14657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:53:16.128393 14657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:53:16.128417 14657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:53:16.129526 14657 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:53:16.129560 14657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:53:16.129575 14657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:53:16.129587 14657 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:53:16.129599 14657 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:53:16.131033 14657 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:53:16.131290 14657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:53:16.131556 14657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:53:16.136459 14665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:16.136477 14657 server_base.cc:1034] running on GCE node
W20260926 20:53:16.136487 14663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:16.136459 14662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:16.136955 14657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:53:16.137300 14657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:53:16.138487 14657 hybrid_clock.cc:648] HybridClock initialized: now 1790455996138479 us; error 45 us; skew 500 ppm
Sep 26 20:53:16 dist-test-slave-hwlj krb5kdc[13589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:53:16.141258 14657 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:53:16.141790 14657 webserver.cc:460] Webserver started at http://127.0.98.62:42341/ using document root <none> and password file <none>
I20260926 20:53:16.142038 14657 fs_manager.cc:359] Metadata directory not provided
I20260926 20:53:16.142107 14657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:53:16.143510 14657 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:53:16.144423 14672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:16.144666 14657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:53:16.144798 14657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "c34ec07649dc4073aacd19b4012322dc"
format_stamp: "Formatted at 2026-09-26 20:53:14 on dist-test-slave-hwlj"
server_key: "616e1915dbee0d3609dbda21e4706069"
server_key_iv: "bd689dc330e0949e0dc7ef90d6824a40"
server_key_version: "encryptionkey@0"
I20260926 20:53:16.145159 14657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:53:16.152136 14657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:53:16.154392 14657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:53:16.154549 14657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:53:16.158562 14657 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40317
I20260926 20:53:16.158594 14724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40317 every 8 connection(s)
I20260926 20:53:16.158988 14657 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:53:16.160571 14725 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:53:16.161213 14725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc: Bootstrap starting.
I20260926 20:53:16.161717 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 14657
I20260926 20:53:16.161998 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:36027
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40317
--webserver_port=32983
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:53:16.170146 14725 log.cc:824] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc: Log is configured to *not* fsync() on all Append() calls
I20260926 20:53:16.172652 14725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:53:16.172952 14725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc: Bootstrap complete.
I20260926 20:53:16.175197 14725 raft_consensus.cc:348] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c34ec07649dc4073aacd19b4012322dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40317 } }
I20260926 20:53:16.175486 14725 raft_consensus.cc:729] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c34ec07649dc4073aacd19b4012322dc, State: Initialized, Role: FOLLOWER
I20260926 20:53:16.175691 14725 consensus_queue.cc:260] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c34ec07649dc4073aacd19b4012322dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40317 } }
I20260926 20:53:16.175788 14725 raft_consensus.cc:388] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:53:16.175840 14725 raft_consensus.cc:482] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:53:16.175894 14725 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc [term 2 FOLLOWER]: Advancing to term 3
I20260926 20:53:16.176936 14725 raft_consensus.cc:504] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c34ec07649dc4073aacd19b4012322dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40317 } }
I20260926 20:53:16.177029 14725 leader_election.cc:302] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c34ec07649dc4073aacd19b4012322dc; no voters:
I20260926 20:53:16.177248 14725 leader_election.cc:288] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 20:53:16.177290 14729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc [term 3 FOLLOWER]: Leader election won for term 3
I20260926 20:53:16.177410 14729 raft_consensus.cc:686] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc [term 3 LEADER]: Becoming Leader. State: Replica: c34ec07649dc4073aacd19b4012322dc, State: Running, Role: LEADER
I20260926 20:53:16.177496 14729 consensus_queue.cc:237] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c34ec07649dc4073aacd19b4012322dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40317 } }
I20260926 20:53:16.177525 14725 sys_catalog.cc:564] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:53:16.178114 14731 sys_catalog.cc:455] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader c34ec07649dc4073aacd19b4012322dc. Latest consensus state: current_term: 3 leader_uuid: "c34ec07649dc4073aacd19b4012322dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c34ec07649dc4073aacd19b4012322dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40317 } } }
I20260926 20:53:16.178234 14731 sys_catalog.cc:458] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc [sys.catalog]: This master's current role is: LEADER
I20260926 20:53:16.178114 14730 sys_catalog.cc:455] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "c34ec07649dc4073aacd19b4012322dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c34ec07649dc4073aacd19b4012322dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40317 } } }
I20260926 20:53:16.178443 14730 sys_catalog.cc:458] T 00000000000000000000000000000000 P c34ec07649dc4073aacd19b4012322dc [sys.catalog]: This master's current role is: LEADER
I20260926 20:53:16.178436 14738 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:53:16.179334 14738 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:53:16.179445 14738 catalog_manager.cc:1259] Loaded cluster ID: 4d31bd1307c440b4ad6e21af55ea2962
I20260926 20:53:16.179478 14738 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 20:53:16.179821 14738 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fad2cce59ad google::LogMessage::Fail() at ??:0
@ 0x7fad2cce888a google::LogMessage::SendToLog() at ??:0
@ 0x7fad2cce54de google::LogMessage::Flush() at ??:0
@ 0x7fad2cce6a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fad34c060f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fad34c0f8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fad2ebb30d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fad2eba8954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fad2fa0c6db start_thread at ??:0
@ 0x7fad2b7e361f clone at ??:0
W20260926 20:53:16.257848 14727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:53:16.258062 14727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:53:16.258088 14727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:53:16.259310 14727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:53:16.259353 14727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:53:16.259382 14727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:53:16.260952 14727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:36027
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=32983
--tserver_master_addrs=127.0.98.62:40317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:53:16.261199 14727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:53:16.261458 14727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:53:16.266619 14762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:16.266606 14761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:16.266604 14764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:16.266943 14727 server_base.cc:1034] running on GCE node
I20260926 20:53:16.267139 14727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:53:16.267354 14727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:53:16.268499 14727 hybrid_clock.cc:648] HybridClock initialized: now 1790455996268486 us; error 30 us; skew 500 ppm
Sep 26 20:53:16 dist-test-slave-hwlj krb5kdc[13589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:53:16.271075 14727 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 20:53:16.271552 14727 webserver.cc:460] Webserver started at http://127.0.98.1:32983/ using document root <none> and password file <none>
I20260926 20:53:16.271858 14727 fs_manager.cc:359] Metadata directory not provided
I20260926 20:53:16.271929 14727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:53:16.273396 14727 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:53:16.274221 14771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:16.274415 14727 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:53:16.274519 14727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-0/wal
uuid: "f8d233a0707a4a69abbe2d682788fd00"
format_stamp: "Formatted at 2026-09-26 20:53:14 on dist-test-slave-hwlj"
server_key: "c2a4d2cd252fc5535b31a01d2e2d17c0"
server_key_iv: "894ea1d3b39725f2819d9ccbb7f96d34"
server_key_version: "encryptionkey@0"
I20260926 20:53:16.274928 14727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:53:16.294799 14727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:53:16.297485 14727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:53:16.297652 14727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:53:16.297909 14727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:53:16.298368 14778 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 20:53:16.300463 14727 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 20:53:16.300534 14727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 20:53:16.300583 14727 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 20:53:16.301236 14727 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 20:53:16.301297 14727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 20:53:16.301371 14778 tablet_bootstrap.cc:492] T b708fc2e4c664be3b876c6ed919b6079 P f8d233a0707a4a69abbe2d682788fd00: Bootstrap starting.
I20260926 20:53:16.308651 14727 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36027
I20260926 20:53:16.308687 14885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36027 every 8 connection(s)
I20260926 20:53:16.309093 14727 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:53:16.312170 14778 log.cc:824] T b708fc2e4c664be3b876c6ed919b6079 P f8d233a0707a4a69abbe2d682788fd00: Log is configured to *not* fsync() on all Append() calls
I20260926 20:53:16.314990 14778 tablet_bootstrap.cc:492] T b708fc2e4c664be3b876c6ed919b6079 P f8d233a0707a4a69abbe2d682788fd00: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:53:16.315333 14778 tablet_bootstrap.cc:492] T b708fc2e4c664be3b876c6ed919b6079 P f8d233a0707a4a69abbe2d682788fd00: Bootstrap complete.
I20260926 20:53:16.316582 14778 ts_tablet_manager.cc:1397] T b708fc2e4c664be3b876c6ed919b6079 P f8d233a0707a4a69abbe2d682788fd00: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.006s
I20260926 20:53:16.317596 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 14727
I20260926 20:53:16.317843 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42061
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40317
--webserver_port=42563
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:53:16.318048 14778 raft_consensus.cc:348] T b708fc2e4c664be3b876c6ed919b6079 P f8d233a0707a4a69abbe2d682788fd00 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86e43b5e22124c6ca7c9ef94ad156f31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42061 } } peers { permanent_uuid: "501a255b1a0a4f6aba703ba67d771eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33619 } } peers { permanent_uuid: "f8d233a0707a4a69abbe2d682788fd00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36027 } }
I20260926 20:53:16.318423 14778 raft_consensus.cc:729] T b708fc2e4c664be3b876c6ed919b6079 P f8d233a0707a4a69abbe2d682788fd00 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8d233a0707a4a69abbe2d682788fd00, State: Initialized, Role: FOLLOWER
I20260926 20:53:16.318578 14778 consensus_queue.cc:260] T b708fc2e4c664be3b876c6ed919b6079 P f8d233a0707a4a69abbe2d682788fd00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86e43b5e22124c6ca7c9ef94ad156f31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42061 } } peers { permanent_uuid: "501a255b1a0a4f6aba703ba67d771eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33619 } } peers { permanent_uuid: "f8d233a0707a4a69abbe2d682788fd00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36027 } }
I20260926 20:53:16.318858 14778 ts_tablet_manager.cc:1428] T b708fc2e4c664be3b876c6ed919b6079 P f8d233a0707a4a69abbe2d682788fd00: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
W20260926 20:53:16.387097 14888 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:53:16.309803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.1:35137)
0926 20:53:16.309993 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:16.310036 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:16.310509 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:16.386891 (+ 76382us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":112,"threads_started":1}
W20260926 20:53:16.387387 14886 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40317 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40317: Client connection negotiation failed: client connection to 127.0.98.62:40317: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 20:53:16.387814 14888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:16.387635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.1:42167)
0926 20:53:16.387688 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:16.387780 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 20:53:16.388157 14888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:16.388071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.1:37749)
0926 20:53:16.388119 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:16.388142 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 20:53:16.388284 14886 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 20:53:16.402750 14891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:53:16.402961 14891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:53:16.402985 14891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:53:16.404261 14891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:53:16.404310 14891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:53:16.404338 14891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:53:16.405867 14891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42061
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=42563
--tserver_master_addrs=127.0.98.62:40317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:53:16.406114 14891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:53:16.406353 14891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:53:16.411479 14896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:16.411551 14891 server_base.cc:1034] running on GCE node
W20260926 20:53:16.411489 14897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:16.411497 14899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:16.411885 14891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:53:16.412153 14891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:53:16.413337 14891 hybrid_clock.cc:648] HybridClock initialized: now 1790455996413317 us; error 53 us; skew 500 ppm
Sep 26 20:53:16 dist-test-slave-hwlj krb5kdc[13589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:53:16.415907 14891 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 20:53:16.416383 14891 webserver.cc:460] Webserver started at http://127.0.98.2:42563/ using document root <none> and password file <none>
I20260926 20:53:16.416574 14891 fs_manager.cc:359] Metadata directory not provided
I20260926 20:53:16.416615 14891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:53:16.417922 14891 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:53:16.418577 14906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:16.418767 14891 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:53:16.418861 14891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-1/wal
uuid: "86e43b5e22124c6ca7c9ef94ad156f31"
format_stamp: "Formatted at 2026-09-26 20:53:15 on dist-test-slave-hwlj"
server_key: "747bee5fe3fbf14f682d2014cea91db2"
server_key_iv: "7ba6bf0d05e6caf0b49dbf2d7cc1d82b"
server_key_version: "encryptionkey@0"
I20260926 20:53:16.419212 14891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:53:16.457880 14891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:53:16.460372 14891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:53:16.460546 14891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:53:16.460801 14891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:53:16.461270 14913 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 20:53:16.462792 14891 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 20:53:16.462872 14891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 20:53:16.462929 14891 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 20:53:16.463685 14891 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 20:53:16.463769 14891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260926 20:53:16.463851 14913 tablet_bootstrap.cc:492] T b708fc2e4c664be3b876c6ed919b6079 P 86e43b5e22124c6ca7c9ef94ad156f31: Bootstrap starting.
I20260926 20:53:16.470369 14891 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42061
I20260926 20:53:16.470451 15020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42061 every 8 connection(s)
I20260926 20:53:16.470772 14891 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
I20260926 20:53:16.472025 15023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:16.471433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.2:56007)
0926 20:53:16.471647 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:16.471798 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":86,"threads_started":1}
W20260926 20:53:16.472252 15021 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40317 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40317: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
I20260926 20:53:16.472568 15023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:16.472465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.2:47681)
0926 20:53:16.472517 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:16.472546 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 20:53:16.472882 15023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:16.472825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.2:33363)
0926 20:53:16.472849 (+ 24us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:16.472868 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260926 20:53:16.473009 15021 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 20:53:16.473627 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 14891
I20260926 20:53:16.473917 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33619
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40317
--webserver_port=43491
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:53:16.475059 14913 log.cc:824] T b708fc2e4c664be3b876c6ed919b6079 P 86e43b5e22124c6ca7c9ef94ad156f31: Log is configured to *not* fsync() on all Append() calls
I20260926 20:53:16.477344 14913 tablet_bootstrap.cc:492] T b708fc2e4c664be3b876c6ed919b6079 P 86e43b5e22124c6ca7c9ef94ad156f31: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:53:16.477694 14913 tablet_bootstrap.cc:492] T b708fc2e4c664be3b876c6ed919b6079 P 86e43b5e22124c6ca7c9ef94ad156f31: Bootstrap complete.
I20260926 20:53:16.478760 14913 ts_tablet_manager.cc:1397] T b708fc2e4c664be3b876c6ed919b6079 P 86e43b5e22124c6ca7c9ef94ad156f31: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.009s
I20260926 20:53:16.480086 14913 raft_consensus.cc:348] T b708fc2e4c664be3b876c6ed919b6079 P 86e43b5e22124c6ca7c9ef94ad156f31 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86e43b5e22124c6ca7c9ef94ad156f31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42061 } } peers { permanent_uuid: "501a255b1a0a4f6aba703ba67d771eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33619 } } peers { permanent_uuid: "f8d233a0707a4a69abbe2d682788fd00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36027 } }
I20260926 20:53:16.480346 14913 raft_consensus.cc:729] T b708fc2e4c664be3b876c6ed919b6079 P 86e43b5e22124c6ca7c9ef94ad156f31 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86e43b5e22124c6ca7c9ef94ad156f31, State: Initialized, Role: FOLLOWER
I20260926 20:53:16.480489 14913 consensus_queue.cc:260] T b708fc2e4c664be3b876c6ed919b6079 P 86e43b5e22124c6ca7c9ef94ad156f31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86e43b5e22124c6ca7c9ef94ad156f31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42061 } } peers { permanent_uuid: "501a255b1a0a4f6aba703ba67d771eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33619 } } peers { permanent_uuid: "f8d233a0707a4a69abbe2d682788fd00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36027 } }
I20260926 20:53:16.480784 14913 ts_tablet_manager.cc:1428] T b708fc2e4c664be3b876c6ed919b6079 P 86e43b5e22124c6ca7c9ef94ad156f31: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 20:53:16.481091 15023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:16.480970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.2:37673)
0926 20:53:16.481026 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:16.481057 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 20:53:16.481458 15023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:16.481328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.2:32851)
0926 20:53:16.481384 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:16.481420 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 20:53:16.559692 15024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:53:16.559994 15024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:53:16.560021 15024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:53:16.561303 15024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:53:16.561357 15024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:53:16.561385 15024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:53:16.562959 15024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33619
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43491
--tserver_master_addrs=127.0.98.62:40317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:53:16.563207 15024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:53:16.563472 15024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:53:16.568630 15032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:16.568626 15034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:16.568670 15031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:16.568806 15024 server_base.cc:1034] running on GCE node
I20260926 20:53:16.569101 15024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:53:16.569381 15024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:53:16.570581 15024 hybrid_clock.cc:648] HybridClock initialized: now 1790455996570570 us; error 37 us; skew 500 ppm
Sep 26 20:53:16 dist-test-slave-hwlj krb5kdc[13589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:53:16.573348 15024 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 20:53:16.573781 15024 webserver.cc:460] Webserver started at http://127.0.98.3:43491/ using document root <none> and password file <none>
I20260926 20:53:16.573995 15024 fs_manager.cc:359] Metadata directory not provided
I20260926 20:53:16.574039 15024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:53:16.575410 15024 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:53:16.576144 15041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:16.576365 15024 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:53:16.576462 15024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-2/wal
uuid: "501a255b1a0a4f6aba703ba67d771eb2"
format_stamp: "Formatted at 2026-09-26 20:53:15 on dist-test-slave-hwlj"
server_key: "b9a42a4e0e3a57bc620d569a5d9a711e"
server_key_iv: "b0ba9d53c134fad71ac78713a3e8243f"
server_key_version: "encryptionkey@0"
I20260926 20:53:16.577013 15024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:53:16.601867 15024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:53:16.604453 15024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:53:16.604580 15024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:53:16.604782 15024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:53:16.605130 15024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:53:16.605168 15024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:16.605191 15024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:53:16.605239 15024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:16.611078 15024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33619
I20260926 20:53:16.611124 15154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33619 every 8 connection(s)
I20260926 20:53:16.611474 15024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
I20260926 20:53:16.612506 15157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:16.612129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.3:35749)
0926 20:53:16.612289 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:16.612364 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":76,"threads_started":1}
W20260926 20:53:16.612900 15155 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40317 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40317: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
I20260926 20:53:16.613217 15157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:16.613122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.3:54137)
0926 20:53:16.613171 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:16.613199 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 20:53:16.613468 15157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:16.613418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.3:34721)
0926 20:53:16.613439 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:16.613458 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260926 20:53:16.613580 15155 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 20:53:16.621315 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 15024
I20260926 20:53:17.388922 15159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:17.388684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.1:49843)
0926 20:53:17.388816 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:17.388857 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":65,"threads_started":1}
I20260926 20:53:17.482093 15160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:17.481879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.2:52543)
0926 20:53:17.481996 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:17.482033 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":57,"threads_started":1}
I20260926 20:53:17.548053 15161 raft_consensus.cc:482] T b708fc2e4c664be3b876c6ed919b6079 P f8d233a0707a4a69abbe2d682788fd00 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:53:17.548189 15161 raft_consensus.cc:504] T b708fc2e4c664be3b876c6ed919b6079 P f8d233a0707a4a69abbe2d682788fd00 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86e43b5e22124c6ca7c9ef94ad156f31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42061 } } peers { permanent_uuid: "501a255b1a0a4f6aba703ba67d771eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33619 } } peers { permanent_uuid: "f8d233a0707a4a69abbe2d682788fd00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36027 } }
I20260926 20:53:17.548473 15161 leader_election.cc:288] T b708fc2e4c664be3b876c6ed919b6079 P f8d233a0707a4a69abbe2d682788fd00 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 86e43b5e22124c6ca7c9ef94ad156f31 (127.0.98.2:42061), 501a255b1a0a4f6aba703ba67d771eb2 (127.0.98.3:33619)
Sep 26 20:53:17 dist-test-slave-hwlj krb5kdc[13589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 26 20:53:17 dist-test-slave-hwlj krb5kdc[13589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260926 20:53:17.558732 15163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:17.548861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56741 (local address 127.0.98.2:42061)
0926 20:53:17.549010 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:17.549021 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:17.549049 (+ 28us) server_negotiation.cc:407] Connection header received
0926 20:53:17.549173 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:17.549174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:17.549214 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:17.549345 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:17.550031 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:17.550920 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:17.552019 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:17.552421 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:17.552550 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:17.555259 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:17.555263 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:17.555264 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:17.555276 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:17.556986 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:17.557386 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:17.557388 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:17.557389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:17.557487 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:17.557721 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:17.557722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:17.557722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:17.558050 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:17.558166 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:17.558481 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 20:53:17.558616 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":88,"threads_started":1}
I20260926 20:53:17.558743 15162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:17.548790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58985 (local address 127.0.98.3:33619)
0926 20:53:17.548949 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:17.548963 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:17.549121 (+ 158us) server_negotiation.cc:407] Connection header received
0926 20:53:17.549597 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:17.549600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:17.549650 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:17.549820 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:17.550158 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:17.551023 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:17.552018 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:17.552418 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:17.552550 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:17.554983 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:17.554988 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:17.554989 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:17.555001 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:17.556986 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:17.557565 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:17.557568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:17.557569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:17.557723 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:17.557861 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:17.557862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:17.557863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:17.558126 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:17.558202 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:17.558481 (+ 279us) server_negotiation.cc:299] Negotiation successful
0926 20:53:17.558655 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":96,"threads_started":1}
I20260926 20:53:17.558771 15159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:17.548660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42061 (local address 127.0.98.1:56741)
0926 20:53:17.548715 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:17.548730 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:17.548824 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:17.549392 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:17.549395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:17.549737 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:17.549919 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:17.549925 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:17.551031 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:17.551032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:17.551934 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:17.551936 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:17.552525 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:17.552526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:17.552637 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:17.553287 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:17.553296 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:17.555159 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:53:17.557078 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:17.557084 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:17.557085 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:17.557296 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:17.557575 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:17.557576 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:17.557576 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:17.557658 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:17.558119 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:53:17.558124 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:53:17.558287 (+ 163us) client_negotiation.cc:769] Sending connection context
0926 20:53:17.558405 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 20:53:17.558617 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":55}
I20260926 20:53:17.558771 15164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:17.548660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33619 (local address 127.0.98.1:58985)
0926 20:53:17.549039 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:17.549056 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:17.549132 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:17.549804 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:17.549805 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:17.550003 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:17.550049 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:17.550051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:17.551118 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:17.551119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:17.551932 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:17.551935 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:17.552524 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:17.552525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:17.552634 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:17.553337 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:17.553342 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:17.554867 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:53:17.557364 (+ 2497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:17.557364 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:17.557364 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:17.557489 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:17.557788 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:17.557788 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:17.557788 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:17.557825 (+ 37us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:17.558199 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:53:17.558200 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:53:17.558287 (+ 87us) client_negotiation.cc:769] Sending connection context
0926 20:53:17.558405 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 20:53:17.558673 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":433,"thread_start_us":46,"threads_started":1}
I20260926 20:53:17.559237 14975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b708fc2e4c664be3b876c6ed919b6079" candidate_uuid: "f8d233a0707a4a69abbe2d682788fd00" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "86e43b5e22124c6ca7c9ef94ad156f31" is_pre_election: true
I20260926 20:53:17.559412 14975 raft_consensus.cc:2454] T b708fc2e4c664be3b876c6ed919b6079 P 86e43b5e22124c6ca7c9ef94ad156f31 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8d233a0707a4a69abbe2d682788fd00 in term 1.
I20260926 20:53:17.559734 14773 leader_election.cc:302] T b708fc2e4c664be3b876c6ed919b6079 P f8d233a0707a4a69abbe2d682788fd00 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86e43b5e22124c6ca7c9ef94ad156f31, f8d233a0707a4a69abbe2d682788fd00; no voters:
I20260926 20:53:17.560083 15161 raft_consensus.cc:2781] T b708fc2e4c664be3b876c6ed919b6079 P f8d233a0707a4a69abbe2d682788fd00 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 20:53:17.560177 15161 raft_consensus.cc:482] T b708fc2e4c664be3b876c6ed919b6079 P f8d233a0707a4a69abbe2d682788fd00 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:53:17.560205 15161 raft_consensus.cc:3037] T b708fc2e4c664be3b876c6ed919b6079 P f8d233a0707a4a69abbe2d682788fd00 [term 1 FOLLOWER]: Advancing to term 2
W20260926 20:53:17.560210 14772 leader_election.cc:341] T b708fc2e4c664be3b876c6ed919b6079 P f8d233a0707a4a69abbe2d682788fd00 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 501a255b1a0a4f6aba703ba67d771eb2 (127.0.98.3:33619): Not found: Tablet not found: b708fc2e4c664be3b876c6ed919b6079
I20260926 20:53:17.561371 15161 raft_consensus.cc:504] T b708fc2e4c664be3b876c6ed919b6079 P f8d233a0707a4a69abbe2d682788fd00 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86e43b5e22124c6ca7c9ef94ad156f31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42061 } } peers { permanent_uuid: "501a255b1a0a4f6aba703ba67d771eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33619 } } peers { permanent_uuid: "f8d233a0707a4a69abbe2d682788fd00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36027 } }
I20260926 20:53:17.561532 15161 leader_election.cc:288] T b708fc2e4c664be3b876c6ed919b6079 P f8d233a0707a4a69abbe2d682788fd00 [CANDIDATE]: Term 2 election: Requested vote from peers 86e43b5e22124c6ca7c9ef94ad156f31 (127.0.98.2:42061), 501a255b1a0a4f6aba703ba67d771eb2 (127.0.98.3:33619)
I20260926 20:53:17.561736 14975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b708fc2e4c664be3b876c6ed919b6079" candidate_uuid: "f8d233a0707a4a69abbe2d682788fd00" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "86e43b5e22124c6ca7c9ef94ad156f31"
W20260926 20:53:17.561825 14772 leader_election.cc:341] T b708fc2e4c664be3b876c6ed919b6079 P f8d233a0707a4a69abbe2d682788fd00 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 501a255b1a0a4f6aba703ba67d771eb2 (127.0.98.3:33619): Not found: Tablet not found: b708fc2e4c664be3b876c6ed919b6079
I20260926 20:53:17.561820 14975 raft_consensus.cc:3037] T b708fc2e4c664be3b876c6ed919b6079 P 86e43b5e22124c6ca7c9ef94ad156f31 [term 1 FOLLOWER]: Advancing to term 2
I20260926 20:53:17.562914 14975 raft_consensus.cc:2454] T b708fc2e4c664be3b876c6ed919b6079 P 86e43b5e22124c6ca7c9ef94ad156f31 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8d233a0707a4a69abbe2d682788fd00 in term 2.
I20260926 20:53:17.563094 14773 leader_election.cc:302] T b708fc2e4c664be3b876c6ed919b6079 P f8d233a0707a4a69abbe2d682788fd00 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 86e43b5e22124c6ca7c9ef94ad156f31, f8d233a0707a4a69abbe2d682788fd00; no voters: 501a255b1a0a4f6aba703ba67d771eb2
I20260926 20:53:17.563189 15161 raft_consensus.cc:2781] T b708fc2e4c664be3b876c6ed919b6079 P f8d233a0707a4a69abbe2d682788fd00 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 20:53:17.563328 15161 raft_consensus.cc:686] T b708fc2e4c664be3b876c6ed919b6079 P f8d233a0707a4a69abbe2d682788fd00 [term 2 LEADER]: Becoming Leader. State: Replica: f8d233a0707a4a69abbe2d682788fd00, State: Running, Role: LEADER
I20260926 20:53:17.563441 15161 consensus_queue.cc:237] T b708fc2e4c664be3b876c6ed919b6079 P f8d233a0707a4a69abbe2d682788fd00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86e43b5e22124c6ca7c9ef94ad156f31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42061 } } peers { permanent_uuid: "501a255b1a0a4f6aba703ba67d771eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33619 } } peers { permanent_uuid: "f8d233a0707a4a69abbe2d682788fd00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36027 } }
I20260926 20:53:17.563707 15164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:17.563600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.1:58967)
0926 20:53:17.563643 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:17.563666 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 20:53:17.614102 15167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:17.613906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.3:40301)
0926 20:53:17.614025 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:17.614069 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":55,"threads_started":1}
W20260926 20:53:17.998863 14772 consensus_peers.cc:487] T b708fc2e4c664be3b876c6ed919b6079 P f8d233a0707a4a69abbe2d682788fd00 -> Peer 501a255b1a0a4f6aba703ba67d771eb2 (127.0.98.3:33619): Couldn't send request to peer 501a255b1a0a4f6aba703ba67d771eb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b708fc2e4c664be3b876c6ed919b6079. This is attempt 1: this message will repeat every 5th retry.
I20260926 20:53:18.001682 14975 raft_consensus.cc:1261] T b708fc2e4c664be3b876c6ed919b6079 P 86e43b5e22124c6ca7c9ef94ad156f31 [term 2 FOLLOWER]: Refusing update from remote peer f8d233a0707a4a69abbe2d682788fd00: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260926 20:53:18.002048 15161 consensus_queue.cc:1035] T b708fc2e4c664be3b876c6ed919b6079 P f8d233a0707a4a69abbe2d682788fd00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86e43b5e22124c6ca7c9ef94ad156f31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260926 20:53:18.003031 15170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:18.002841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.2:41221)
0926 20:53:18.002939 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:18.002975 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":59,"threads_started":1}
I20260926 20:53:18.003799 15164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:18.003642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.1:34093)
0926 20:53:18.003699 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:18.003738 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 20:53:18.004110 15170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:18.003999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.2:43497)
0926 20:53:18.004063 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:18.004089 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 20:53:18.551119 15173 ts_tablet_manager.cc:927] T b708fc2e4c664be3b876c6ed919b6079 P 501a255b1a0a4f6aba703ba67d771eb2: Initiating tablet copy from peer f8d233a0707a4a69abbe2d682788fd00 (127.0.98.1:36027)
I20260926 20:53:18.551362 15173 tablet_copy_client.cc:316] T b708fc2e4c664be3b876c6ed919b6079 P 501a255b1a0a4f6aba703ba67d771eb2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.1:36027
Sep 26 20:53:18 dist-test-slave-hwlj krb5kdc[13589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 20:53:18.558918 15175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:18.551684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36027 (local address 127.0.98.3:33421)
0926 20:53:18.551873 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:18.551890 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:18.551954 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:18.552104 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:18.552107 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:18.552397 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:18.552457 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:18.552459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:18.553309 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:18.553309 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:18.553875 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:18.553877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:18.554325 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:18.554326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:18.554377 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:18.554819 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:18.554824 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:18.556233 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:53:18.557954 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:18.557959 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:18.557961 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:18.558156 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:18.558346 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:18.558346 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:18.558346 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:18.558400 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:18.558637 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:53:18.558643 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:53:18.558686 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 20:53:18.558756 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 20:53:18.558841 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":129,"threads_started":1}
I20260926 20:53:18.558921 15176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:18.551819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33421 (local address 127.0.98.1:36027)
0926 20:53:18.551960 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:18.551965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:18.551975 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:53:18.552000 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:18.552002 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:18.552014 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:18.552100 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:18.552529 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:18.553226 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:18.553946 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:18.554257 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:18.554315 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:18.556318 (+ 2003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:18.556327 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:18.556328 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:18.556336 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:18.557877 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:18.558231 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:18.558234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:18.558234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:18.558290 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:18.558450 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:18.558450 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:18.558450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:18.558585 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:18.558645 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:18.558793 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 20:53:18.558855 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":50,"threads_started":1}
I20260926 20:53:18.559369 14860 tablet_copy_service.cc:140] P f8d233a0707a4a69abbe2d682788fd00: Received BeginTabletCopySession request for tablet b708fc2e4c664be3b876c6ed919b6079 from peer 501a255b1a0a4f6aba703ba67d771eb2 ({username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33421)
I20260926 20:53:18.559433 14860 tablet_copy_service.cc:161] P f8d233a0707a4a69abbe2d682788fd00: Beginning new tablet copy session on tablet b708fc2e4c664be3b876c6ed919b6079 from peer 501a255b1a0a4f6aba703ba67d771eb2 at {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33421: session id = 501a255b1a0a4f6aba703ba67d771eb2-b708fc2e4c664be3b876c6ed919b6079
I20260926 20:53:18.559937 14860 tablet_copy_source_session.cc:205] T b708fc2e4c664be3b876c6ed919b6079 P f8d233a0707a4a69abbe2d682788fd00: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 20:53:18.560832 15173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b708fc2e4c664be3b876c6ed919b6079. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:53:18.563370 15173 tablet_copy_client.cc:791] T b708fc2e4c664be3b876c6ed919b6079 P 501a255b1a0a4f6aba703ba67d771eb2: tablet copy: Starting download of 0 data blocks...
I20260926 20:53:18.563525 15173 tablet_copy_client.cc:655] T b708fc2e4c664be3b876c6ed919b6079 P 501a255b1a0a4f6aba703ba67d771eb2: tablet copy: Starting download of 1 WAL segments...
I20260926 20:53:18.564466 15173 tablet_copy_client.cc:531] T b708fc2e4c664be3b876c6ed919b6079 P 501a255b1a0a4f6aba703ba67d771eb2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 20:53:18.565116 15173 tablet_bootstrap.cc:492] T b708fc2e4c664be3b876c6ed919b6079 P 501a255b1a0a4f6aba703ba67d771eb2: Bootstrap starting.
I20260926 20:53:18.566419 15173 log.cc:824] T b708fc2e4c664be3b876c6ed919b6079 P 501a255b1a0a4f6aba703ba67d771eb2: Log is configured to *not* fsync() on all Append() calls
I20260926 20:53:18.569017 15173 tablet_bootstrap.cc:492] T b708fc2e4c664be3b876c6ed919b6079 P 501a255b1a0a4f6aba703ba67d771eb2: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:53:18.569209 15173 tablet_bootstrap.cc:492] T b708fc2e4c664be3b876c6ed919b6079 P 501a255b1a0a4f6aba703ba67d771eb2: Bootstrap complete.
I20260926 20:53:18.569306 15173 ts_tablet_manager.cc:1397] T b708fc2e4c664be3b876c6ed919b6079 P 501a255b1a0a4f6aba703ba67d771eb2: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 20:53:18.570232 15173 raft_consensus.cc:348] T b708fc2e4c664be3b876c6ed919b6079 P 501a255b1a0a4f6aba703ba67d771eb2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86e43b5e22124c6ca7c9ef94ad156f31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42061 } } peers { permanent_uuid: "501a255b1a0a4f6aba703ba67d771eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33619 } } peers { permanent_uuid: "f8d233a0707a4a69abbe2d682788fd00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36027 } }
I20260926 20:53:18.570374 15173 raft_consensus.cc:729] T b708fc2e4c664be3b876c6ed919b6079 P 501a255b1a0a4f6aba703ba67d771eb2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 501a255b1a0a4f6aba703ba67d771eb2, State: Initialized, Role: FOLLOWER
I20260926 20:53:18.570504 15173 consensus_queue.cc:260] T b708fc2e4c664be3b876c6ed919b6079 P 501a255b1a0a4f6aba703ba67d771eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86e43b5e22124c6ca7c9ef94ad156f31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42061 } } peers { permanent_uuid: "501a255b1a0a4f6aba703ba67d771eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33619 } } peers { permanent_uuid: "f8d233a0707a4a69abbe2d682788fd00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36027 } }
I20260926 20:53:18.570811 15173 ts_tablet_manager.cc:1428] T b708fc2e4c664be3b876c6ed919b6079 P 501a255b1a0a4f6aba703ba67d771eb2: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20260926 20:53:18.571100 15175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:18.570979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.3:49735)
0926 20:53:18.571031 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:18.571063 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 20:53:18.571100 14860 tablet_copy_service.cc:342] P f8d233a0707a4a69abbe2d682788fd00: Request end of tablet copy session 501a255b1a0a4f6aba703ba67d771eb2-b708fc2e4c664be3b876c6ed919b6079 received from {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33421
I20260926 20:53:18.571187 14860 tablet_copy_service.cc:433] P f8d233a0707a4a69abbe2d682788fd00: ending tablet copy session 501a255b1a0a4f6aba703ba67d771eb2-b708fc2e4c664be3b876c6ed919b6079 on tablet b708fc2e4c664be3b876c6ed919b6079 with peer 501a255b1a0a4f6aba703ba67d771eb2
I20260926 20:53:18.571435 15175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:18.571341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.3:38459)
0926 20:53:18.571389 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:18.571414 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 20:53:19.004419 15179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:19.004204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.1:48729)
0926 20:53:19.004320 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:19.004356 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":58,"threads_started":1}
I20260926 20:53:19.004840 15180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:19.004638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.2:51723)
0926 20:53:19.004769 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:19.004805 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":69,"threads_started":1}
I20260926 20:53:19.119434 15179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:19.119299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.1:50529)
0926 20:53:19.119360 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:19.119395 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 20:53:19.119480 15109 raft_consensus.cc:1203] T b708fc2e4c664be3b876c6ed919b6079 P 501a255b1a0a4f6aba703ba67d771eb2 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260926 20:53:19.120249 15183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:19.120041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.3:45377)
0926 20:53:19.120170 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:19.120200 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":73,"threads_started":1}
I20260926 20:53:20.005621 15184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:20.005381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.2:43439)
0926 20:53:20.005518 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:20.005573 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":65,"threads_started":1}
I20260926 20:53:20.120216 15185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:20.119977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.1:51223)
0926 20:53:20.120125 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:20.120167 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":63,"threads_started":1}
I20260926 20:53:20.120934 15186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:20.120706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.3:46225)
0926 20:53:20.120821 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:20.120879 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":89,"threads_started":1}
I20260926 20:53:21.006424 15187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:21.006192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.2:50051)
0926 20:53:21.006334 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:21.006377 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":71,"threads_started":1}
I20260926 20:53:21.121076 15188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:21.120814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.1:47903)
0926 20:53:21.120968 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:21.121022 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":79,"threads_started":1}
I20260926 20:53:21.121737 15189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:21.121467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.3:50199)
0926 20:53:21.121655 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:21.121700 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":128,"threads_started":1}
I20260926 20:53:22.007211 15190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:22.006988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.2:60191)
0926 20:53:22.007122 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:22.007166 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":66,"threads_started":1}
I20260926 20:53:22.121898 15191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:22.121687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.1:60583)
0926 20:53:22.121815 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:22.121854 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":60,"threads_started":1}
I20260926 20:53:22.122308 15191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:22.122192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.1:46171)
0926 20:53:22.122248 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:22.122279 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 20:53:22.122539 15192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:22.122315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.3:34151)
0926 20:53:22.122457 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:22.122491 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":71,"threads_started":1}
I20260926 20:53:23.007926 15193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:23.007697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.2:38693)
0926 20:53:23.007845 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:23.007880 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":89,"threads_started":1}
I20260926 20:53:23.123023 15194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:23.122807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.1:52133)
0926 20:53:23.122938 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:23.122974 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":67,"threads_started":1}
I20260926 20:53:23.123200 15195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:23.122957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.3:59539)
0926 20:53:23.123084 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:23.123140 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":87,"threads_started":1}
I20260926 20:53:24.008687 15196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:24.008473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.2:54991)
0926 20:53:24.008600 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:24.008641 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":63,"threads_started":1}
I20260926 20:53:24.123839 15197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:24.123545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.1:46319)
0926 20:53:24.123699 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:24.123733 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":81,"threads_started":1}
I20260926 20:53:24.123867 15198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:24.123646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.3:56979)
0926 20:53:24.123785 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:24.123818 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":78,"threads_started":1}
I20260926 20:53:25.009406 15199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:25.009195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.2:56975)
0926 20:53:25.009323 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:25.009360 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":67,"threads_started":1}
I20260926 20:53:25.124586 15201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:25.124360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.1:57251)
0926 20:53:25.124497 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:25.124537 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":78,"threads_started":1}
I20260926 20:53:25.124598 15200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:25.124360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.3:57627)
0926 20:53:25.124525 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:25.124557 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":78,"threads_started":1}
I20260926 20:53:26.010238 15202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:26.009930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.2:47439)
0926 20:53:26.010119 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:26.010156 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":75,"threads_started":1}
I20260926 20:53:26.125355 15203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:26.125103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.1:40677)
0926 20:53:26.125252 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:26.125292 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":77,"threads_started":1}
I20260926 20:53:26.125501 15204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:26.125183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.3:36247)
0926 20:53:26.125400 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:26.125447 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":83,"threads_started":1}
I20260926 20:53:27.010988 15205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:27.010762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.2:42007)
0926 20:53:27.010905 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:27.010943 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":71,"threads_started":1}
I20260926 20:53:27.126062 15206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:27.125836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.1:38551)
0926 20:53:27.125984 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:27.126016 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":89,"threads_started":1}
I20260926 20:53:27.126099 15207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:27.125907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.3:59965)
0926 20:53:27.126023 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:27.126059 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":73,"threads_started":1}
I20260926 20:53:28.011880 15208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:28.011561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.2:57125)
0926 20:53:28.011742 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:28.011827 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":90,"threads_started":1}
I20260926 20:53:28.126822 15210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:28.126607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.3:33273)
0926 20:53:28.126723 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:28.126769 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":47,"threads_started":1}
I20260926 20:53:28.126822 15209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:28.126575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.1:54151)
0926 20:53:28.126723 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:28.126768 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":74,"threads_started":1}
I20260926 20:53:29.012564 15211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:29.012366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.2:55857)
0926 20:53:29.012483 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:29.012518 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":60,"threads_started":1}
I20260926 20:53:29.127583 15213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:29.127357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.1:46139)
0926 20:53:29.127503 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:29.127542 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":94,"threads_started":1}
I20260926 20:53:29.127574 15212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:29.127356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.3:57911)
0926 20:53:29.127491 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:29.127530 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":76,"threads_started":1}
I20260926 20:53:30.013280 15214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:30.013039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.2:57517)
0926 20:53:30.013196 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:30.013236 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":73,"threads_started":1}
I20260926 20:53:30.128357 15216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:30.128134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.3:49815)
0926 20:53:30.128264 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:30.128303 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":57,"threads_started":1}
I20260926 20:53:30.128403 15217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:30.128179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.1:47535)
0926 20:53:30.128328 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:30.128361 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":86,"threads_started":1}
I20260926 20:53:31.014026 15218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:31.013795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.2:38915)
0926 20:53:31.013941 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:31.013979 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":75,"threads_started":1}
I20260926 20:53:31.129070 15220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:31.128811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.3:33577)
0926 20:53:31.128999 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:31.129031 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":104,"threads_started":1}
I20260926 20:53:31.129061 15219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:31.128810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40317 (local address 127.0.98.1:43659)
0926 20:53:31.128964 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:31.129004 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":103,"threads_started":1}
I20260926 20:53:31.629741 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 14727
I20260926 20:53:31.634550 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 14891
I20260926 20:53:31.639269 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 15024
2026-09-26T20:53:31Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18182 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:53:31 dist-test-slave-hwlj krb5kdc[15223](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:53:31 dist-test-slave-hwlj krb5kdc[15223](info): set up 2 sockets
Sep 26 20:53:31 dist-test-slave-hwlj krb5kdc[15223](info): commencing operation
krb5kdc: starting...
W20260926 20:53:32.679860 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/test-user.keytab.
Sep 26 20:53:32 dist-test-slave-hwlj krb5kdc[15223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:53:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:53:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 20:53:32.799594 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43565
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:53:32.882251 15236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:53:32.882525 15236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:53:32.882552 15236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:53:32.884012 15236 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 20:53:32.884060 15236 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 20:53:32.884075 15236 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 20:53:32.884089 15236 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:53:32.884104 15236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:53:32.884156 15236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:53:32.884177 15236 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:53:32.884202 15236 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:53:32.885900 15236 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:53:32.886183 15236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:53:32.886444 15236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:53:32.892163 15236 server_base.cc:1034] running on GCE node
W20260926 20:53:32.892154 15242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:32.892114 15244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:32.892114 15241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:32.892598 15236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:53:32.892889 15236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:53:32.894068 15236 hybrid_clock.cc:648] HybridClock initialized: now 1790456012894045 us; error 45 us; skew 500 ppm
Sep 26 20:53:32 dist-test-slave-hwlj krb5kdc[15223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:53:32.900663 15236 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:53:32.901172 15236 webserver.cc:460] Webserver started at http://127.0.98.62:39329/ using document root <none> and password file <none>
I20260926 20:53:32.901401 15236 fs_manager.cc:359] Metadata directory not provided
I20260926 20:53:32.901469 15236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:53:32.901629 15236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:53:32.902712 15236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/master-0/data/instance:
uuid: "eddc2af3af4e422187c40b7416070efe"
format_stamp: "Formatted at 2026-09-26 20:53:32 on dist-test-slave-hwlj"
server_key: "06a0ca64fe3568ad0f5001e1dd534c9c"
server_key_iv: "dcbfedebbc2015f628b518203f1b07fc"
server_key_version: "encryptionkey@0"
I20260926 20:53:32.903081 15236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/master-0/wal/instance:
uuid: "eddc2af3af4e422187c40b7416070efe"
format_stamp: "Formatted at 2026-09-26 20:53:32 on dist-test-slave-hwlj"
server_key: "06a0ca64fe3568ad0f5001e1dd534c9c"
server_key_iv: "dcbfedebbc2015f628b518203f1b07fc"
server_key_version: "encryptionkey@0"
I20260926 20:53:32.904487 15236 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 20:53:32.905241 15251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:32.905406 15236 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:53:32.905529 15236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "eddc2af3af4e422187c40b7416070efe"
format_stamp: "Formatted at 2026-09-26 20:53:32 on dist-test-slave-hwlj"
server_key: "06a0ca64fe3568ad0f5001e1dd534c9c"
server_key_iv: "dcbfedebbc2015f628b518203f1b07fc"
server_key_version: "encryptionkey@0"
I20260926 20:53:32.905606 15236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:53:32.935067 15236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:53:32.937304 15236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:53:32.937518 15236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:53:32.942233 15236 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43565
I20260926 20:53:32.942269 15303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43565 every 8 connection(s)
I20260926 20:53:32.942698 15236 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:53:32.943578 15304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:53:32.944763 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 15236
I20260926 20:53:32.944895 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/master-0/wal/instance
I20260926 20:53:32.945103 392 external_mini_cluster.cc:1428] Setting key 2c8ae04ed41f4287257a2bcbf77966b6
I20260926 20:53:32.946439 15304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe: Bootstrap starting.
I20260926 20:53:32.947165 15304 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe: Neither blocks nor log segments found. Creating new log.
I20260926 20:53:32.947520 15304 log.cc:824] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe: Log is configured to *not* fsync() on all Append() calls
I20260926 20:53:32.948237 15304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe: No bootstrap required, opened a new log
I20260926 20:53:32.950557 15304 raft_consensus.cc:348] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eddc2af3af4e422187c40b7416070efe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43565 } }
I20260926 20:53:32.950735 15304 raft_consensus.cc:374] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:53:32.950775 15304 raft_consensus.cc:729] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eddc2af3af4e422187c40b7416070efe, State: Initialized, Role: FOLLOWER
I20260926 20:53:32.950917 15304 consensus_queue.cc:260] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eddc2af3af4e422187c40b7416070efe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43565 } }
I20260926 20:53:32.951004 15304 raft_consensus.cc:388] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:53:32.951040 15304 raft_consensus.cc:482] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:53:32.951092 15304 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:53:32.951920 15304 raft_consensus.cc:504] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eddc2af3af4e422187c40b7416070efe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43565 } }
I20260926 20:53:32.952010 15304 leader_election.cc:302] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eddc2af3af4e422187c40b7416070efe; no voters:
I20260926 20:53:32.952206 15304 leader_election.cc:288] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:53:32.952241 15309 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:53:32.952358 15309 raft_consensus.cc:686] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe [term 1 LEADER]: Becoming Leader. State: Replica: eddc2af3af4e422187c40b7416070efe, State: Running, Role: LEADER
I20260926 20:53:32.952500 15304 sys_catalog.cc:564] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe [sys.catalog]: configured and running, proceeding with master startup.
Sep 26 20:53:32 dist-test-slave-hwlj krb5kdc[15223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:53:32.952500 15309 consensus_queue.cc:237] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eddc2af3af4e422187c40b7416070efe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43565 } }
I20260926 20:53:32.952917 15309 sys_catalog.cc:455] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe [sys.catalog]: SysCatalogTable state changed. Reason: New leader eddc2af3af4e422187c40b7416070efe. Latest consensus state: current_term: 1 leader_uuid: "eddc2af3af4e422187c40b7416070efe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eddc2af3af4e422187c40b7416070efe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43565 } } }
I20260926 20:53:32.952998 15309 sys_catalog.cc:458] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe [sys.catalog]: This master's current role is: LEADER
I20260926 20:53:32.952931 15311 sys_catalog.cc:455] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eddc2af3af4e422187c40b7416070efe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eddc2af3af4e422187c40b7416070efe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43565 } } }
I20260926 20:53:32.953037 15311 sys_catalog.cc:458] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe [sys.catalog]: This master's current role is: LEADER
I20260926 20:53:32.953593 15315 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:53:32.954137 15315 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:53:32.956162 15315 catalog_manager.cc:1347] Generated new cluster ID: f1e596f2c2db459a95634083ca1d1205
I20260926 20:53:32.956213 15315 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:53:32.957206 15307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:32.945661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33244 (local address 127.0.98.62:43565)
0926 20:53:32.945862 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:32.945881 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:32.945916 (+ 35us) server_negotiation.cc:407] Connection header received
0926 20:53:32.946430 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:32.946440 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:32.946500 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:32.946641 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:32.947614 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:32.948658 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:32.949723 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:32.950124 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:32.950204 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:32.953504 (+ 3300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:32.953508 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:32.953509 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:32.953523 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:32.955596 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:32.956015 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:32.956018 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:32.956018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:32.956132 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:32.956351 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:32.956352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:32.956352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:32.956585 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:32.956733 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:32.956844 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 20:53:32.956996 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":100,"threads_started":1}
I20260926 20:53:32.993691 15315 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:53:32.994246 15315 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:53:32.997014 15315 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe: Generated new TSK 0
I20260926 20:53:32.997251 15315 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 20:53:33.023154 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43565
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:53:33.105101 15330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:53:33.105325 15330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:53:33.105373 15330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:53:33.106561 15330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:53:33.106634 15330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:53:33.106688 15330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:53:33.108300 15330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:53:33.108568 15330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:53:33.108788 15330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:53:33.113992 15338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:33.114012 15336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:33.114008 15335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:33.114100 15330 server_base.cc:1034] running on GCE node
I20260926 20:53:33.114483 15330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:53:33.114804 15330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:53:33.115983 15330 hybrid_clock.cc:648] HybridClock initialized: now 1790456013115971 us; error 41 us; skew 500 ppm
Sep 26 20:53:33 dist-test-slave-hwlj krb5kdc[15223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:53:33.118775 15330 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 20:53:33.119288 15330 webserver.cc:460] Webserver started at http://127.0.98.1:36875/ using document root <none> and password file <none>
I20260926 20:53:33.119524 15330 fs_manager.cc:359] Metadata directory not provided
I20260926 20:53:33.119594 15330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:53:33.119793 15330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:53:33.120937 15330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-0/data/instance:
uuid: "9a0a759577424f46b7341380af800802"
format_stamp: "Formatted at 2026-09-26 20:53:33 on dist-test-slave-hwlj"
server_key: "ee6e85469a0994cf33e261b7008e0ed3"
server_key_iv: "e5e60cc76a891fff44870d8472b99adc"
server_key_version: "encryptionkey@0"
I20260926 20:53:33.121310 15330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9a0a759577424f46b7341380af800802"
format_stamp: "Formatted at 2026-09-26 20:53:33 on dist-test-slave-hwlj"
server_key: "ee6e85469a0994cf33e261b7008e0ed3"
server_key_iv: "e5e60cc76a891fff44870d8472b99adc"
server_key_version: "encryptionkey@0"
I20260926 20:53:33.122786 15330 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:53:33.123605 15345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:33.123896 15330 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:53:33.124029 15330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-0/wal
uuid: "9a0a759577424f46b7341380af800802"
format_stamp: "Formatted at 2026-09-26 20:53:33 on dist-test-slave-hwlj"
server_key: "ee6e85469a0994cf33e261b7008e0ed3"
server_key_iv: "e5e60cc76a891fff44870d8472b99adc"
server_key_version: "encryptionkey@0"
I20260926 20:53:33.124125 15330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:53:33.143379 15330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:53:33.145884 15330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:53:33.146080 15330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:53:33.146333 15330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:53:33.146651 15330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:53:33.146705 15330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:33.146746 15330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:53:33.146775 15330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:33.152776 15330 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41173
I20260926 20:53:33.152822 15458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41173 every 8 connection(s)
I20260926 20:53:33.153179 15330 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:53:33.158466 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 15330
I20260926 20:53:33.158550 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-0/wal/instance
I20260926 20:53:33.158771 392 external_mini_cluster.cc:1428] Setting key c444af6cb023bee519c84b9d2aa424f9
Sep 26 20:53:33 dist-test-slave-hwlj krb5kdc[15223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:53:33.163218 15461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:33.154000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.1:49337)
0926 20:53:33.154179 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:33.154216 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:33.154706 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:33.155007 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:33.155010 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:33.155587 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:33.155798 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:33.155802 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:33.156449 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:33.156449 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:33.157236 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:33.157238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:33.157611 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:33.157612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:33.157707 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:33.158156 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:33.158163 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:33.159724 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:53:33.161666 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:33.161674 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:33.161677 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:33.162025 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:33.162265 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:33.162266 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:33.162266 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:33.162422 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:33.162721 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:53:33.162727 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:53:33.162844 (+ 117us) client_negotiation.cc:769] Sending connection context
0926 20:53:33.162934 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 20:53:33.163061 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":103,"threads_started":1}
I20260926 20:53:33.163239 15307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:33.154036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49337 (local address 127.0.98.62:43565)
0926 20:53:33.154111 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:33.154114 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:33.154296 (+ 182us) server_negotiation.cc:407] Connection header received
0926 20:53:33.154867 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:33.154869 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:33.154882 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:33.154942 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:33.155871 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:33.156363 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:33.157309 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:33.157536 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:33.157592 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:33.159846 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:33.159847 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:33.159848 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:33.159855 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:33.161566 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:33.162120 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:33.162122 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:33.162122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:33.162186 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:33.162494 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:33.162495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:33.162495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:33.162645 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:33.162716 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:33.162985 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 20:53:33.163094 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 20:53:33.163798 15459 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43565
I20260926 20:53:33.163903 15459 heartbeater.cc:464] Registering TS with master...
I20260926 20:53:33.164121 15459 heartbeater.cc:511] Master 127.0.98.62:43565 requested a full tablet report, sending...
I20260926 20:53:33.164672 15268 ts_manager.cc:194] Registered new tserver with Master: 9a0a759577424f46b7341380af800802 (127.0.98.1:41173)
I20260926 20:53:33.165328 15268 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49337
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 20:53:33.186558 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43565
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:53:33.269691 15464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:53:33.269938 15464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:53:33.269980 15464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:53:33.271132 15464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:53:33.271202 15464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:53:33.271248 15464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:53:33.272735 15464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:53:33.273005 15464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:53:33.273233 15464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:53:33.279989 15472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:33.279989 15470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:33.280196 15469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:33.280174 15464 server_base.cc:1034] running on GCE node
I20260926 20:53:33.280476 15464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:53:33.280717 15464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:53:33.281867 15464 hybrid_clock.cc:648] HybridClock initialized: now 1790456013281854 us; error 27 us; skew 500 ppm
Sep 26 20:53:33 dist-test-slave-hwlj krb5kdc[15223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:53:33.284535 15464 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 20:53:33.284968 15464 webserver.cc:460] Webserver started at http://127.0.98.2:46305/ using document root <none> and password file <none>
I20260926 20:53:33.285214 15464 fs_manager.cc:359] Metadata directory not provided
I20260926 20:53:33.285260 15464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:53:33.285384 15464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:53:33.286453 15464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-1/data/instance:
uuid: "fb366842fb274b3fa8b268cece62eab0"
format_stamp: "Formatted at 2026-09-26 20:53:33 on dist-test-slave-hwlj"
server_key: "d3dde8a914d4b2b7e8e057f788e4ab75"
server_key_iv: "f98a5af4f03254e4fe9e351b774e53f4"
server_key_version: "encryptionkey@0"
I20260926 20:53:33.286813 15464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fb366842fb274b3fa8b268cece62eab0"
format_stamp: "Formatted at 2026-09-26 20:53:33 on dist-test-slave-hwlj"
server_key: "d3dde8a914d4b2b7e8e057f788e4ab75"
server_key_iv: "f98a5af4f03254e4fe9e351b774e53f4"
server_key_version: "encryptionkey@0"
I20260926 20:53:33.288250 15464 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:53:33.289021 15479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:33.289227 15464 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:53:33.289318 15464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-1/wal
uuid: "fb366842fb274b3fa8b268cece62eab0"
format_stamp: "Formatted at 2026-09-26 20:53:33 on dist-test-slave-hwlj"
server_key: "d3dde8a914d4b2b7e8e057f788e4ab75"
server_key_iv: "f98a5af4f03254e4fe9e351b774e53f4"
server_key_version: "encryptionkey@0"
I20260926 20:53:33.289390 15464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:53:33.311511 15464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:53:33.315806 15464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:53:33.315997 15464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:53:33.316231 15464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:53:33.316563 15464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:53:33.316618 15464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:33.316674 15464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:53:33.316704 15464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:33.322449 15464 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41677
I20260926 20:53:33.322525 15592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41677 every 8 connection(s)
I20260926 20:53:33.322860 15464 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 20:53:33 dist-test-slave-hwlj krb5kdc[15223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:53:33.331823 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 15464
I20260926 20:53:33.331914 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-1/wal/instance
I20260926 20:53:33.332123 392 external_mini_cluster.cc:1428] Setting key f9f7c2833efe989dc2ca7ddda2ce815f
I20260926 20:53:33.332226 15307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:33.323609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49353 (local address 127.0.98.62:43565)
0926 20:53:33.323687 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:33.323690 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:33.323897 (+ 207us) server_negotiation.cc:407] Connection header received
0926 20:53:33.324388 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:33.324390 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:33.324400 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:33.324467 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:33.325353 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:33.325900 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:33.326930 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:33.327130 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:33.327198 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:33.329364 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:33.329366 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:33.329366 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:33.329372 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:33.330790 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:33.331158 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:33.331159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:33.331159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:33.331219 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:33.331473 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:33.331474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:33.331474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:33.331598 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:33.331672 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:33.332047 (+ 375us) server_negotiation.cc:299] Negotiation successful
0926 20:53:33.332120 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 20:53:33.332684 15595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:33.323556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.2:49353)
0926 20:53:33.323730 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:33.323808 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:33.324283 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:33.324521 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:33.324524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:33.325078 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:33.325263 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:33.325267 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:33.325980 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:33.325981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:33.326769 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:33.326770 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:33.327205 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:33.327206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:33.327313 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:33.327833 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:33.327842 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:33.329279 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:53:33.330861 (+ 1582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:33.330867 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:33.330868 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:33.331078 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:33.331286 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:33.331286 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:33.331287 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:33.331378 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:33.331675 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:53:33.331681 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:53:33.331871 (+ 190us) client_negotiation.cc:769] Sending connection context
0926 20:53:33.332029 (+ 158us) client_negotiation.cc:240] Negotiation successful
0926 20:53:33.332219 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":89,"threads_started":1}
I20260926 20:53:33.333133 15593 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43565
I20260926 20:53:33.333226 15593 heartbeater.cc:464] Registering TS with master...
I20260926 20:53:33.333446 15593 heartbeater.cc:511] Master 127.0.98.62:43565 requested a full tablet report, sending...
I20260926 20:53:33.333813 15268 ts_manager.cc:194] Registered new tserver with Master: fb366842fb274b3fa8b268cece62eab0 (127.0.98.2:41677)
I20260926 20:53:33.334324 15268 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49353
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 20:53:33.358799 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43565
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:53:33.440847 15598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:53:33.441052 15598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:53:33.441077 15598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:53:33.442233 15598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:53:33.442313 15598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:53:33.442340 15598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:53:33.443892 15598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:53:33.444149 15598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:53:33.444398 15598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:53:33.449339 15603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:33.449345 15604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:33.449337 15606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:33.449548 15598 server_base.cc:1034] running on GCE node
I20260926 20:53:33.449748 15598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:53:33.450006 15598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:53:33.451174 15598 hybrid_clock.cc:648] HybridClock initialized: now 1790456013451135 us; error 55 us; skew 500 ppm
Sep 26 20:53:33 dist-test-slave-hwlj krb5kdc[15223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:53:33.453917 15598 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 20:53:33.454506 15598 webserver.cc:460] Webserver started at http://127.0.98.3:41589/ using document root <none> and password file <none>
I20260926 20:53:33.454780 15598 fs_manager.cc:359] Metadata directory not provided
I20260926 20:53:33.454859 15598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:53:33.454993 15598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:53:33.456140 15598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-2/data/instance:
uuid: "1788a70030a64ae7b5059ac6a1700692"
format_stamp: "Formatted at 2026-09-26 20:53:33 on dist-test-slave-hwlj"
server_key: "dfbb201cea7e108042799806bed68557"
server_key_iv: "894f56d5750433eda96deba6a9e04248"
server_key_version: "encryptionkey@0"
I20260926 20:53:33.456503 15598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1788a70030a64ae7b5059ac6a1700692"
format_stamp: "Formatted at 2026-09-26 20:53:33 on dist-test-slave-hwlj"
server_key: "dfbb201cea7e108042799806bed68557"
server_key_iv: "894f56d5750433eda96deba6a9e04248"
server_key_version: "encryptionkey@0"
I20260926 20:53:33.458051 15598 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:53:33.458864 15613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:33.459064 15598 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:53:33.459163 15598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-2/wal
uuid: "1788a70030a64ae7b5059ac6a1700692"
format_stamp: "Formatted at 2026-09-26 20:53:33 on dist-test-slave-hwlj"
server_key: "dfbb201cea7e108042799806bed68557"
server_key_iv: "894f56d5750433eda96deba6a9e04248"
server_key_version: "encryptionkey@0"
I20260926 20:53:33.459247 15598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:53:33.475729 15598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:53:33.484799 15598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:53:33.484967 15598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:53:33.485220 15598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:53:33.485615 15598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:53:33.485671 15598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:33.485714 15598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:53:33.485741 15598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:33.492825 15598 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40147
I20260926 20:53:33.492892 15726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40147 every 8 connection(s)
I20260926 20:53:33.493201 15598 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
I20260926 20:53:33.493803 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 15598
I20260926 20:53:33.493903 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-2/wal/instance
I20260926 20:53:33.494128 392 external_mini_cluster.cc:1428] Setting key f5910a36c0543aaa6853b22c94fcaf7d
Sep 26 20:53:33 dist-test-slave-hwlj krb5kdc[15223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:53:33.503722 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:33.493939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.3:36691)
0926 20:53:33.494131 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:33.494181 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:33.494942 (+ 761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:33.495185 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:33.495189 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:33.495741 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:33.496001 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:33.496006 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:33.497141 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:33.497143 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:33.497962 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:33.497965 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:33.498522 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:33.498523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:33.498622 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:33.499179 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:33.499188 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:33.500512 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:53:33.502348 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:33.502354 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:33.502356 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:33.502596 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:33.502817 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:33.502818 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:33.502818 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:33.502917 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:33.503237 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:53:33.503242 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:53:33.503358 (+ 116us) client_negotiation.cc:769] Sending connection context
0926 20:53:33.503446 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 20:53:33.503563 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":90,"threads_started":1}
I20260926 20:53:33.503813 15307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:33.494237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36691 (local address 127.0.98.62:43565)
0926 20:53:33.494284 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:33.494286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:33.494300 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:53:33.495047 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:33.495048 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:33.495062 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:33.495115 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:33.496084 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:33.497053 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:33.498047 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:33.498424 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:33.498515 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:33.500604 (+ 2089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:33.500605 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:33.500606 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:33.500614 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:33.502268 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:33.502677 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:33.502678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:33.502679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:33.502743 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:33.502997 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:33.502997 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:33.502998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:33.503151 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:33.503230 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:33.503547 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 20:53:33.503644 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 20:53:33.504274 15727 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43565
I20260926 20:53:33.504387 15727 heartbeater.cc:464] Registering TS with master...
I20260926 20:53:33.504565 15727 heartbeater.cc:511] Master 127.0.98.62:43565 requested a full tablet report, sending...
I20260926 20:53:33.504962 15268 ts_manager.cc:194] Registered new tserver with Master: 1788a70030a64ae7b5059ac6a1700692 (127.0.98.3:40147)
I20260926 20:53:33.505544 15268 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36691
I20260926 20:53:33.505857 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:53:33.513666 15307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:33.507342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33256 (local address 127.0.98.62:43565)
0926 20:53:33.507379 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:33.507380 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:33.507515 (+ 135us) server_negotiation.cc:407] Connection header received
0926 20:53:33.507621 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:33.507623 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:33.507633 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:33.507681 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:33.508289 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:33.508823 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:33.509548 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:33.509758 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:33.509833 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:33.511043 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:33.511044 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:33.511045 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:33.511052 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:33.512502 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:33.512856 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:33.512857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:33.512857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:33.512919 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:33.513120 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:33.513121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:33.513121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:33.513271 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:33.513375 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:33.513477 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 20:53:33.513569 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 20:53:33.515334 15268 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33256:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:53:33.516983 15268 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:53:33.524529 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:33.519396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41173 (local address 127.0.0.1:37262)
0926 20:53:33.519943 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:33.519988 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:33.520059 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:33.520422 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:33.520426 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:33.520429 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:53:33.520494 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:33.520496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:33.521828 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:33.521829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:33.523175 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:33.523178 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:33.524229 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:33.524230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:33.524313 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:33.524326 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 20:53:33.524391 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 20:53:33.524455 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":28,"thread_start_us":117,"threads_started":1}
I20260926 20:53:33.524874 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:33.519894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37262 (local address 127.0.98.1:41173)
0926 20:53:33.520172 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:33.520177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:33.520189 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:53:33.520229 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:33.520231 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:33.520248 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:33.520325 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:53:33.520574 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:33.521723 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:33.523282 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:33.524044 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:33.524127 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:33.524191 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:33.524734 (+ 543us) server_negotiation.cc:299] Negotiation successful
0926 20:53:33.524779 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":97,"threads_started":1}
I20260926 20:53:33.525455 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:33.519396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40147 (local address 127.0.0.1:60086)
0926 20:53:33.520167 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:33.520178 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:33.520257 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:33.520501 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:33.520501 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:33.520503 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:53:33.520546 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:33.520547 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:33.522304 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:33.522305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:33.523714 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:33.523717 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:33.525173 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:33.525174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:33.525254 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:33.525259 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:53:33.525328 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 20:53:33.525371 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"mutex_wait_us":2,"thread_start_us":60,"threads_started":1}
I20260926 20:53:33.525612 15741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:33.519656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60086 (local address 127.0.98.3:40147)
0926 20:53:33.519859 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:33.519864 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:33.520230 (+ 366us) server_negotiation.cc:407] Connection header received
0926 20:53:33.520347 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:33.520349 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:33.520362 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:33.520411 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:53:33.520641 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:33.521882 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:33.523852 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:33.525074 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:33.525158 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:33.525221 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:33.525346 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 20:53:33.525392 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":98,"threads_started":1}
I20260926 20:53:33.526266 15393 tablet_service.cc:1467] Processing CreateTablet for tablet 0c74fe380fdf407a8fd25cef6e5382c2 (DEFAULT_TABLE table=test-table [id=24ff8b191b684491aa49d6171977f194]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:53:33.526274 15661 tablet_service.cc:1467] Processing CreateTablet for tablet 0c74fe380fdf407a8fd25cef6e5382c2 (DEFAULT_TABLE table=test-table [id=24ff8b191b684491aa49d6171977f194]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:53:33.526379 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:33.519437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41677 (local address 127.0.0.1:56454)
0926 20:53:33.520654 (+ 1217us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:33.520663 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:33.520704 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:33.520957 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:33.520958 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:33.520960 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:53:33.521000 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:33.521001 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:33.522185 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:33.522186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:33.523314 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:33.523317 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:33.524230 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:33.524231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:33.524313 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:33.524326 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 20:53:33.524391 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 20:53:33.526313 (+ 1922us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1200,"mutex_wait_us":1923,"thread_start_us":106,"threads_started":1}
I20260926 20:53:33.526652 15661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c74fe380fdf407a8fd25cef6e5382c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:53:33.526654 15393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c74fe380fdf407a8fd25cef6e5382c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:53:33.526767 15742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:33.519672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56454 (local address 127.0.98.2:41677)
0926 20:53:33.519880 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:33.519885 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:33.520760 (+ 875us) server_negotiation.cc:407] Connection header received
0926 20:53:33.520804 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:33.520806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:33.520821 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:33.520887 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:53:33.521066 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:33.522080 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:33.523394 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:33.524100 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:33.524154 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:33.524200 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:33.524439 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 20:53:33.524474 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":108,"threads_started":1}
I20260926 20:53:33.527318 15527 tablet_service.cc:1467] Processing CreateTablet for tablet 0c74fe380fdf407a8fd25cef6e5382c2 (DEFAULT_TABLE table=test-table [id=24ff8b191b684491aa49d6171977f194]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:53:33.527632 15527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c74fe380fdf407a8fd25cef6e5382c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:53:33.530758 15745 tablet_bootstrap.cc:492] T 0c74fe380fdf407a8fd25cef6e5382c2 P fb366842fb274b3fa8b268cece62eab0: Bootstrap starting.
I20260926 20:53:33.531653 15745 tablet_bootstrap.cc:654] T 0c74fe380fdf407a8fd25cef6e5382c2 P fb366842fb274b3fa8b268cece62eab0: Neither blocks nor log segments found. Creating new log.
I20260926 20:53:33.531353 15746 tablet_bootstrap.cc:492] T 0c74fe380fdf407a8fd25cef6e5382c2 P 1788a70030a64ae7b5059ac6a1700692: Bootstrap starting.
I20260926 20:53:33.532001 15745 log.cc:824] T 0c74fe380fdf407a8fd25cef6e5382c2 P fb366842fb274b3fa8b268cece62eab0: Log is configured to *not* fsync() on all Append() calls
I20260926 20:53:33.532408 15746 tablet_bootstrap.cc:654] T 0c74fe380fdf407a8fd25cef6e5382c2 P 1788a70030a64ae7b5059ac6a1700692: Neither blocks nor log segments found. Creating new log.
I20260926 20:53:33.532719 15745 tablet_bootstrap.cc:492] T 0c74fe380fdf407a8fd25cef6e5382c2 P fb366842fb274b3fa8b268cece62eab0: No bootstrap required, opened a new log
I20260926 20:53:33.532800 15745 ts_tablet_manager.cc:1397] T 0c74fe380fdf407a8fd25cef6e5382c2 P fb366842fb274b3fa8b268cece62eab0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:53:33.533066 15746 log.cc:824] T 0c74fe380fdf407a8fd25cef6e5382c2 P 1788a70030a64ae7b5059ac6a1700692: Log is configured to *not* fsync() on all Append() calls
I20260926 20:53:33.533282 15744 tablet_bootstrap.cc:492] T 0c74fe380fdf407a8fd25cef6e5382c2 P 9a0a759577424f46b7341380af800802: Bootstrap starting.
I20260926 20:53:33.534204 15744 tablet_bootstrap.cc:654] T 0c74fe380fdf407a8fd25cef6e5382c2 P 9a0a759577424f46b7341380af800802: Neither blocks nor log segments found. Creating new log.
I20260926 20:53:33.534509 15745 raft_consensus.cc:348] T 0c74fe380fdf407a8fd25cef6e5382c2 P fb366842fb274b3fa8b268cece62eab0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a0a759577424f46b7341380af800802" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41173 } } peers { permanent_uuid: "1788a70030a64ae7b5059ac6a1700692" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40147 } } peers { permanent_uuid: "fb366842fb274b3fa8b268cece62eab0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41677 } }
I20260926 20:53:33.534633 15746 tablet_bootstrap.cc:492] T 0c74fe380fdf407a8fd25cef6e5382c2 P 1788a70030a64ae7b5059ac6a1700692: No bootstrap required, opened a new log
I20260926 20:53:33.534677 15745 raft_consensus.cc:374] T 0c74fe380fdf407a8fd25cef6e5382c2 P fb366842fb274b3fa8b268cece62eab0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:53:33.534699 15746 ts_tablet_manager.cc:1397] T 0c74fe380fdf407a8fd25cef6e5382c2 P 1788a70030a64ae7b5059ac6a1700692: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 20:53:33.534736 15745 raft_consensus.cc:729] T 0c74fe380fdf407a8fd25cef6e5382c2 P fb366842fb274b3fa8b268cece62eab0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb366842fb274b3fa8b268cece62eab0, State: Initialized, Role: FOLLOWER
I20260926 20:53:33.534909 15745 consensus_queue.cc:260] T 0c74fe380fdf407a8fd25cef6e5382c2 P fb366842fb274b3fa8b268cece62eab0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a0a759577424f46b7341380af800802" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41173 } } peers { permanent_uuid: "1788a70030a64ae7b5059ac6a1700692" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40147 } } peers { permanent_uuid: "fb366842fb274b3fa8b268cece62eab0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41677 } }
I20260926 20:53:33.535199 15745 ts_tablet_manager.cc:1428] T 0c74fe380fdf407a8fd25cef6e5382c2 P fb366842fb274b3fa8b268cece62eab0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:53:33.535326 15593 heartbeater.cc:503] Master 127.0.98.62:43565 was elected leader, sending a full tablet report...
I20260926 20:53:33.536247 15744 log.cc:824] T 0c74fe380fdf407a8fd25cef6e5382c2 P 9a0a759577424f46b7341380af800802: Log is configured to *not* fsync() on all Append() calls
I20260926 20:53:33.536943 15744 tablet_bootstrap.cc:492] T 0c74fe380fdf407a8fd25cef6e5382c2 P 9a0a759577424f46b7341380af800802: No bootstrap required, opened a new log
I20260926 20:53:33.537025 15744 ts_tablet_manager.cc:1397] T 0c74fe380fdf407a8fd25cef6e5382c2 P 9a0a759577424f46b7341380af800802: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260926 20:53:33.538318 15746 raft_consensus.cc:348] T 0c74fe380fdf407a8fd25cef6e5382c2 P 1788a70030a64ae7b5059ac6a1700692 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a0a759577424f46b7341380af800802" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41173 } } peers { permanent_uuid: "1788a70030a64ae7b5059ac6a1700692" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40147 } } peers { permanent_uuid: "fb366842fb274b3fa8b268cece62eab0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41677 } }
I20260926 20:53:33.538383 15744 raft_consensus.cc:348] T 0c74fe380fdf407a8fd25cef6e5382c2 P 9a0a759577424f46b7341380af800802 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a0a759577424f46b7341380af800802" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41173 } } peers { permanent_uuid: "1788a70030a64ae7b5059ac6a1700692" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40147 } } peers { permanent_uuid: "fb366842fb274b3fa8b268cece62eab0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41677 } }
I20260926 20:53:33.538561 15746 raft_consensus.cc:374] T 0c74fe380fdf407a8fd25cef6e5382c2 P 1788a70030a64ae7b5059ac6a1700692 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:53:33.538589 15744 raft_consensus.cc:374] T 0c74fe380fdf407a8fd25cef6e5382c2 P 9a0a759577424f46b7341380af800802 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:53:33.538607 15746 raft_consensus.cc:729] T 0c74fe380fdf407a8fd25cef6e5382c2 P 1788a70030a64ae7b5059ac6a1700692 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1788a70030a64ae7b5059ac6a1700692, State: Initialized, Role: FOLLOWER
I20260926 20:53:33.538659 15744 raft_consensus.cc:729] T 0c74fe380fdf407a8fd25cef6e5382c2 P 9a0a759577424f46b7341380af800802 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a0a759577424f46b7341380af800802, State: Initialized, Role: FOLLOWER
I20260926 20:53:33.538780 15744 consensus_queue.cc:260] T 0c74fe380fdf407a8fd25cef6e5382c2 P 9a0a759577424f46b7341380af800802 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a0a759577424f46b7341380af800802" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41173 } } peers { permanent_uuid: "1788a70030a64ae7b5059ac6a1700692" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40147 } } peers { permanent_uuid: "fb366842fb274b3fa8b268cece62eab0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41677 } }
I20260926 20:53:33.538766 15746 consensus_queue.cc:260] T 0c74fe380fdf407a8fd25cef6e5382c2 P 1788a70030a64ae7b5059ac6a1700692 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a0a759577424f46b7341380af800802" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41173 } } peers { permanent_uuid: "1788a70030a64ae7b5059ac6a1700692" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40147 } } peers { permanent_uuid: "fb366842fb274b3fa8b268cece62eab0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41677 } }
I20260926 20:53:33.539081 15746 ts_tablet_manager.cc:1428] T 0c74fe380fdf407a8fd25cef6e5382c2 P 1788a70030a64ae7b5059ac6a1700692: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260926 20:53:33.539089 15744 ts_tablet_manager.cc:1428] T 0c74fe380fdf407a8fd25cef6e5382c2 P 9a0a759577424f46b7341380af800802: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 20:53:33.539110 15459 heartbeater.cc:503] Master 127.0.98.62:43565 was elected leader, sending a full tablet report...
I20260926 20:53:33.539214 15727 heartbeater.cc:503] Master 127.0.98.62:43565 was elected leader, sending a full tablet report...
W20260926 20:53:33.573421 15594 tablet.cc:2369] T 0c74fe380fdf407a8fd25cef6e5382c2 P fb366842fb274b3fa8b268cece62eab0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:53:33.647370 15751 raft_consensus.cc:482] T 0c74fe380fdf407a8fd25cef6e5382c2 P 9a0a759577424f46b7341380af800802 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:53:33.647492 15751 raft_consensus.cc:504] T 0c74fe380fdf407a8fd25cef6e5382c2 P 9a0a759577424f46b7341380af800802 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a0a759577424f46b7341380af800802" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41173 } } peers { permanent_uuid: "1788a70030a64ae7b5059ac6a1700692" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40147 } } peers { permanent_uuid: "fb366842fb274b3fa8b268cece62eab0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41677 } }
I20260926 20:53:33.647989 15751 leader_election.cc:288] T 0c74fe380fdf407a8fd25cef6e5382c2 P 9a0a759577424f46b7341380af800802 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1788a70030a64ae7b5059ac6a1700692 (127.0.98.3:40147), fb366842fb274b3fa8b268cece62eab0 (127.0.98.2:41677)
I20260926 20:53:33.650976 15742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:33.648276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59067 (local address 127.0.98.2:41677)
0926 20:53:33.648326 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:33.648328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:33.648375 (+ 47us) server_negotiation.cc:407] Connection header received
0926 20:53:33.648464 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:33.648465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:33.648478 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:33.648548 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:53:33.648734 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:33.649218 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:33.650248 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:33.650629 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:33.650714 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:33.650725 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:33.650877 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 20:53:33.650913 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 20:53:33.650986 15753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:33.648172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41677 (local address 127.0.98.1:59067)
0926 20:53:33.648312 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:33.648324 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:33.648388 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:33.648584 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:33.648585 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:33.648587 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:53:33.648657 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:33.648659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:33.649305 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:33.649306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:33.650153 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:33.650155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:33.650716 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:33.650717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:33.650793 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:33.650797 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:53:33.650859 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 20:53:33.650902 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":52,"thread_start_us":76,"threads_started":1}
I20260926 20:53:33.651057 15741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:33.648364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45941 (local address 127.0.98.3:40147)
0926 20:53:33.648420 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:33.648422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:33.648430 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:53:33.648465 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:33.648466 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:33.648476 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:33.648548 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:53:33.648744 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:33.649255 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:33.650262 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:33.650668 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:33.650767 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:33.650776 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:33.650943 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 20:53:33.650978 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 20:53:33.651098 15461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:33.648157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40147 (local address 127.0.98.1:45941)
0926 20:53:33.648259 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:33.648273 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:33.648328 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:33.648552 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:33.648553 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:33.648555 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:53:33.648659 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:33.648661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:33.649357 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:33.649358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:33.650164 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:33.650166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:33.650769 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:33.650770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:33.650854 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:33.650857 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:53:33.650925 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 20:53:33.650977 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"mutex_wait_us":17}
I20260926 20:53:33.651398 15547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c74fe380fdf407a8fd25cef6e5382c2" candidate_uuid: "9a0a759577424f46b7341380af800802" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb366842fb274b3fa8b268cece62eab0" is_pre_election: true
I20260926 20:53:33.651422 15681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c74fe380fdf407a8fd25cef6e5382c2" candidate_uuid: "9a0a759577424f46b7341380af800802" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1788a70030a64ae7b5059ac6a1700692" is_pre_election: true
I20260926 20:53:33.651559 15547 raft_consensus.cc:2454] T 0c74fe380fdf407a8fd25cef6e5382c2 P fb366842fb274b3fa8b268cece62eab0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a0a759577424f46b7341380af800802 in term 0.
I20260926 20:53:33.651607 15681 raft_consensus.cc:2454] T 0c74fe380fdf407a8fd25cef6e5382c2 P 1788a70030a64ae7b5059ac6a1700692 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a0a759577424f46b7341380af800802 in term 0.
I20260926 20:53:33.651866 15347 leader_election.cc:302] T 0c74fe380fdf407a8fd25cef6e5382c2 P 9a0a759577424f46b7341380af800802 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1788a70030a64ae7b5059ac6a1700692, 9a0a759577424f46b7341380af800802; no voters:
I20260926 20:53:33.652076 15751 raft_consensus.cc:2781] T 0c74fe380fdf407a8fd25cef6e5382c2 P 9a0a759577424f46b7341380af800802 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:53:33.652146 15751 raft_consensus.cc:482] T 0c74fe380fdf407a8fd25cef6e5382c2 P 9a0a759577424f46b7341380af800802 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:53:33.652184 15751 raft_consensus.cc:3037] T 0c74fe380fdf407a8fd25cef6e5382c2 P 9a0a759577424f46b7341380af800802 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:53:33.652971 15751 raft_consensus.cc:504] T 0c74fe380fdf407a8fd25cef6e5382c2 P 9a0a759577424f46b7341380af800802 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a0a759577424f46b7341380af800802" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41173 } } peers { permanent_uuid: "1788a70030a64ae7b5059ac6a1700692" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40147 } } peers { permanent_uuid: "fb366842fb274b3fa8b268cece62eab0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41677 } }
I20260926 20:53:33.653118 15751 leader_election.cc:288] T 0c74fe380fdf407a8fd25cef6e5382c2 P 9a0a759577424f46b7341380af800802 [CANDIDATE]: Term 1 election: Requested vote from peers 1788a70030a64ae7b5059ac6a1700692 (127.0.98.3:40147), fb366842fb274b3fa8b268cece62eab0 (127.0.98.2:41677)
I20260926 20:53:33.653249 15547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c74fe380fdf407a8fd25cef6e5382c2" candidate_uuid: "9a0a759577424f46b7341380af800802" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb366842fb274b3fa8b268cece62eab0"
I20260926 20:53:33.653273 15681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c74fe380fdf407a8fd25cef6e5382c2" candidate_uuid: "9a0a759577424f46b7341380af800802" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1788a70030a64ae7b5059ac6a1700692"
I20260926 20:53:33.653327 15547 raft_consensus.cc:3037] T 0c74fe380fdf407a8fd25cef6e5382c2 P fb366842fb274b3fa8b268cece62eab0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:53:33.653337 15681 raft_consensus.cc:3037] T 0c74fe380fdf407a8fd25cef6e5382c2 P 1788a70030a64ae7b5059ac6a1700692 [term 0 FOLLOWER]: Advancing to term 1
W20260926 20:53:33.653893 15460 tablet.cc:2369] T 0c74fe380fdf407a8fd25cef6e5382c2 P 9a0a759577424f46b7341380af800802: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:53:33.654191 15681 raft_consensus.cc:2454] T 0c74fe380fdf407a8fd25cef6e5382c2 P 1788a70030a64ae7b5059ac6a1700692 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a0a759577424f46b7341380af800802 in term 1.
I20260926 20:53:33.654191 15547 raft_consensus.cc:2454] T 0c74fe380fdf407a8fd25cef6e5382c2 P fb366842fb274b3fa8b268cece62eab0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a0a759577424f46b7341380af800802 in term 1.
I20260926 20:53:33.654369 15347 leader_election.cc:302] T 0c74fe380fdf407a8fd25cef6e5382c2 P 9a0a759577424f46b7341380af800802 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1788a70030a64ae7b5059ac6a1700692, 9a0a759577424f46b7341380af800802; no voters:
I20260926 20:53:33.654467 15751 raft_consensus.cc:2781] T 0c74fe380fdf407a8fd25cef6e5382c2 P 9a0a759577424f46b7341380af800802 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:53:33.654640 15751 raft_consensus.cc:686] T 0c74fe380fdf407a8fd25cef6e5382c2 P 9a0a759577424f46b7341380af800802 [term 1 LEADER]: Becoming Leader. State: Replica: 9a0a759577424f46b7341380af800802, State: Running, Role: LEADER
I20260926 20:53:33.654744 15751 consensus_queue.cc:237] T 0c74fe380fdf407a8fd25cef6e5382c2 P 9a0a759577424f46b7341380af800802 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a0a759577424f46b7341380af800802" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41173 } } peers { permanent_uuid: "1788a70030a64ae7b5059ac6a1700692" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40147 } } peers { permanent_uuid: "fb366842fb274b3fa8b268cece62eab0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41677 } }
I20260926 20:53:33.655570 15267 catalog_manager.cc:5495] T 0c74fe380fdf407a8fd25cef6e5382c2 P 9a0a759577424f46b7341380af800802 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a0a759577424f46b7341380af800802 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "9a0a759577424f46b7341380af800802" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a0a759577424f46b7341380af800802" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1788a70030a64ae7b5059ac6a1700692" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb366842fb274b3fa8b268cece62eab0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41677 } health_report { overall_health: UNKNOWN } } }
I20260926 20:53:33.669304 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:33.666563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37278 (local address 127.0.98.1:41173)
0926 20:53:33.666645 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:33.666647 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:33.666655 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:53:33.666692 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:33.666693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:33.666720 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:33.666805 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:53:33.666960 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:33.667475 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:33.668396 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:33.668639 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:33.668725 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:33.668921 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:53:33.669040 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:53:33.669133 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:33.669182 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 20:53:33.669202 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 20:53:33.671235 15681 raft_consensus.cc:1261] T 0c74fe380fdf407a8fd25cef6e5382c2 P 1788a70030a64ae7b5059ac6a1700692 [term 1 FOLLOWER]: Refusing update from remote peer 9a0a759577424f46b7341380af800802: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:53:33.671285 15547 raft_consensus.cc:1261] T 0c74fe380fdf407a8fd25cef6e5382c2 P fb366842fb274b3fa8b268cece62eab0 [term 1 FOLLOWER]: Refusing update from remote peer 9a0a759577424f46b7341380af800802: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:53:33.671653 15754 consensus_queue.cc:1035] T 0c74fe380fdf407a8fd25cef6e5382c2 P 9a0a759577424f46b7341380af800802 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb366842fb274b3fa8b268cece62eab0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:53:33.671830 15751 consensus_queue.cc:1035] T 0c74fe380fdf407a8fd25cef6e5382c2 P 9a0a759577424f46b7341380af800802 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1788a70030a64ae7b5059ac6a1700692" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:53:33.674266 15757 mvcc.cc:205] Tried to move back new op lower bound from 7333707831994425344 to 7333707831930388480. Current Snapshot: MvccSnapshot[applied={T|T < 7333707831994425344}]
I20260926 20:53:33.674852 15759 mvcc.cc:205] Tried to move back new op lower bound from 7333707831994425344 to 7333707831930388480. Current Snapshot: MvccSnapshot[applied={T|T < 7333707831994425344}]
I20260926 20:53:33.675230 15758 mvcc.cc:205] Tried to move back new op lower bound from 7333707831994425344 to 7333707831930388480. Current Snapshot: MvccSnapshot[applied={T|T < 7333707831994425344 or (T in {7333707831994425344})}]
I20260926 20:53:33.676606 15266 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:53:33.676714 15266 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:53:33.677603 15266 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe: Sending DeleteTablet for 3 replicas of tablet 0c74fe380fdf407a8fd25cef6e5382c2
I20260926 20:53:33.677922 15393 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c74fe380fdf407a8fd25cef6e5382c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:53:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37262
I20260926 20:53:33.677951 15527 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c74fe380fdf407a8fd25cef6e5382c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:53:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56454
I20260926 20:53:33.678292 15767 tablet_replica.cc:330] T 0c74fe380fdf407a8fd25cef6e5382c2 P fb366842fb274b3fa8b268cece62eab0: stopping tablet replica
I20260926 20:53:33.678288 15661 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c74fe380fdf407a8fd25cef6e5382c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:53:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60086
I20260926 20:53:33.678328 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 15330
I20260926 20:53:33.679035 15769 tablet_replica.cc:330] T 0c74fe380fdf407a8fd25cef6e5382c2 P 1788a70030a64ae7b5059ac6a1700692: stopping tablet replica
I20260926 20:53:33.679093 15767 raft_consensus.cc:2229] T 0c74fe380fdf407a8fd25cef6e5382c2 P fb366842fb274b3fa8b268cece62eab0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:53:33.679140 15769 raft_consensus.cc:2229] T 0c74fe380fdf407a8fd25cef6e5382c2 P 1788a70030a64ae7b5059ac6a1700692 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:53:33.679231 15767 raft_consensus.cc:2258] T 0c74fe380fdf407a8fd25cef6e5382c2 P fb366842fb274b3fa8b268cece62eab0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:53:33.679250 15769 raft_consensus.cc:2258] T 0c74fe380fdf407a8fd25cef6e5382c2 P 1788a70030a64ae7b5059ac6a1700692 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:53:33.679715 15767 ts_tablet_manager.cc:1905] T 0c74fe380fdf407a8fd25cef6e5382c2 P fb366842fb274b3fa8b268cece62eab0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:53:33.679728 15769 ts_tablet_manager.cc:1905] T 0c74fe380fdf407a8fd25cef6e5382c2 P 1788a70030a64ae7b5059ac6a1700692: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:53:33.681540 15769 ts_tablet_manager.cc:1918] T 0c74fe380fdf407a8fd25cef6e5382c2 P 1788a70030a64ae7b5059ac6a1700692: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:53:33.681641 15769 log.cc:1196] T 0c74fe380fdf407a8fd25cef6e5382c2 P 1788a70030a64ae7b5059ac6a1700692: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-2/wal/wals/0c74fe380fdf407a8fd25cef6e5382c2
I20260926 20:53:33.681639 15767 ts_tablet_manager.cc:1918] T 0c74fe380fdf407a8fd25cef6e5382c2 P fb366842fb274b3fa8b268cece62eab0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:53:33.681705 15767 log.cc:1196] T 0c74fe380fdf407a8fd25cef6e5382c2 P fb366842fb274b3fa8b268cece62eab0: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-1/wal/wals/0c74fe380fdf407a8fd25cef6e5382c2
I20260926 20:53:33.681954 15769 ts_tablet_manager.cc:1939] T 0c74fe380fdf407a8fd25cef6e5382c2 P 1788a70030a64ae7b5059ac6a1700692: Deleting consensus metadata
I20260926 20:53:33.681991 15767 ts_tablet_manager.cc:1939] T 0c74fe380fdf407a8fd25cef6e5382c2 P fb366842fb274b3fa8b268cece62eab0: Deleting consensus metadata
I20260926 20:53:33.682439 15253 catalog_manager.cc:4841] TS 1788a70030a64ae7b5059ac6a1700692 (127.0.98.3:40147): tablet 0c74fe380fdf407a8fd25cef6e5382c2 (table test-table [id=24ff8b191b684491aa49d6171977f194]) successfully deleted
I20260926 20:53:33.682446 15254 catalog_manager.cc:4841] TS fb366842fb274b3fa8b268cece62eab0 (127.0.98.2:41677): tablet 0c74fe380fdf407a8fd25cef6e5382c2 (table test-table [id=24ff8b191b684491aa49d6171977f194]) successfully deleted
W20260926 20:53:33.684917 15255 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41173 (error 108)
I20260926 20:53:33.685194 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 15464
I20260926 20:53:33.685418 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:33.685301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41173 (local address 127.0.0.1:37294)
0926 20:53:33.685348 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:33.685380 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 20:53:33.686739 15255 catalog_manager.cc:4584] TS 9a0a759577424f46b7341380af800802 (127.0.98.1:41173): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0c74fe380fdf407a8fd25cef6e5382c2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41173: connect: Connection refused (error 111)
I20260926 20:53:33.690235 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 15598
I20260926 20:53:33.694643 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 15236
I20260926 20:53:53.700541 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43565
--webserver_interface=127.0.98.62
--webserver_port=39329
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:53:53.783488 15771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:53:53.783711 15771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:53:53.783735 15771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:53:53.784902 15771 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 20:53:53.784938 15771 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 20:53:53.784951 15771 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 20:53:53.785006 15771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:53:53.785030 15771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:53:53.785063 15771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:53:53.785084 15771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:53:53.785111 15771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:53:53.786578 15771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=39329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:53:53.786774 15771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:53:53.786939 15771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:53:53.795926 15780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:53.795975 15778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:53.795956 15777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:53.796021 15771 server_base.cc:1034] running on GCE node
I20260926 20:53:53.796533 15771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:53:53.796857 15771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:53:53.798134 15771 hybrid_clock.cc:648] HybridClock initialized: now 1790456033798115 us; error 38 us; skew 500 ppm
Sep 26 20:53:53 dist-test-slave-hwlj krb5kdc[15223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:53:53.807138 15771 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:53:53.807670 15771 webserver.cc:460] Webserver started at http://127.0.98.62:39329/ using document root <none> and password file <none>
I20260926 20:53:53.807967 15771 fs_manager.cc:359] Metadata directory not provided
I20260926 20:53:53.808056 15771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:53:53.809463 15771 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:53:53.810223 15787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:53.810407 15771 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:53:53.810531 15771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "eddc2af3af4e422187c40b7416070efe"
format_stamp: "Formatted at 2026-09-26 20:53:32 on dist-test-slave-hwlj"
server_key: "06a0ca64fe3568ad0f5001e1dd534c9c"
server_key_iv: "dcbfedebbc2015f628b518203f1b07fc"
server_key_version: "encryptionkey@0"
I20260926 20:53:53.810978 15771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:53:53.835927 15771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:53:53.837898 15771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:53:53.838073 15771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:53:53.842321 15771 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43565
I20260926 20:53:53.842355 15839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43565 every 8 connection(s)
I20260926 20:53:53.842765 15771 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:53:53.844410 15840 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:53:53.845093 15840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe: Bootstrap starting.
I20260926 20:53:53.845997 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 15771
I20260926 20:53:53.846290 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41173
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43565
--webserver_port=36875
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:53:53.854172 15840 log.cc:824] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe: Log is configured to *not* fsync() on all Append() calls
I20260926 20:53:53.856385 15840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:53:53.856710 15840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe: Bootstrap complete.
I20260926 20:53:53.859119 15840 raft_consensus.cc:348] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eddc2af3af4e422187c40b7416070efe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43565 } }
I20260926 20:53:53.859351 15840 raft_consensus.cc:729] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eddc2af3af4e422187c40b7416070efe, State: Initialized, Role: FOLLOWER
I20260926 20:53:53.859517 15840 consensus_queue.cc:260] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eddc2af3af4e422187c40b7416070efe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43565 } }
I20260926 20:53:53.859622 15840 raft_consensus.cc:388] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:53:53.859683 15840 raft_consensus.cc:482] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:53:53.859741 15840 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe [term 1 FOLLOWER]: Advancing to term 2
I20260926 20:53:53.860831 15840 raft_consensus.cc:504] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eddc2af3af4e422187c40b7416070efe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43565 } }
I20260926 20:53:53.860919 15840 leader_election.cc:302] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eddc2af3af4e422187c40b7416070efe; no voters:
I20260926 20:53:53.861145 15840 leader_election.cc:288] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 20:53:53.861186 15844 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe [term 2 FOLLOWER]: Leader election won for term 2
I20260926 20:53:53.861351 15844 raft_consensus.cc:686] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe [term 2 LEADER]: Becoming Leader. State: Replica: eddc2af3af4e422187c40b7416070efe, State: Running, Role: LEADER
I20260926 20:53:53.861421 15840 sys_catalog.cc:564] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:53:53.861488 15844 consensus_queue.cc:237] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eddc2af3af4e422187c40b7416070efe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43565 } }
I20260926 20:53:53.861805 15844 sys_catalog.cc:455] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe [sys.catalog]: SysCatalogTable state changed. Reason: New leader eddc2af3af4e422187c40b7416070efe. Latest consensus state: current_term: 2 leader_uuid: "eddc2af3af4e422187c40b7416070efe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eddc2af3af4e422187c40b7416070efe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43565 } } }
I20260926 20:53:53.862066 15844 sys_catalog.cc:458] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe [sys.catalog]: This master's current role is: LEADER
I20260926 20:53:53.861985 15845 sys_catalog.cc:455] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "eddc2af3af4e422187c40b7416070efe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eddc2af3af4e422187c40b7416070efe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43565 } } }
I20260926 20:53:53.862414 15845 sys_catalog.cc:458] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe [sys.catalog]: This master's current role is: LEADER
I20260926 20:53:53.862754 15853 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:53:53.863628 15853 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:53:53.863763 15853 catalog_manager.cc:1259] Loaded cluster ID: f1e596f2c2db459a95634083ca1d1205
I20260926 20:53:53.863843 15853 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:53:53.864652 15853 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:53:53.864847 15853 catalog_manager.cc:5879] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe: Loaded TSK: 0
I20260926 20:53:53.874135 15853 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe: Generated new TSK 1
I20260926 20:53:53.874248 15853 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 20:53:53.945785 15842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:53:53.946053 15842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:53:53.946096 15842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:53:53.947309 15842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:53:53.947388 15842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:53:53.947435 15842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:53:53.948998 15842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41173
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36875
--tserver_master_addrs=127.0.98.62:43565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:53:53.949270 15842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:53:53.949519 15842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:53:53.955555 15867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:53.955605 15868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:53.955833 15870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:53.955885 15842 server_base.cc:1034] running on GCE node
I20260926 20:53:53.956079 15842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:53:53.956360 15842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:53:53.957517 15842 hybrid_clock.cc:648] HybridClock initialized: now 1790456033957492 us; error 46 us; skew 500 ppm
Sep 26 20:53:53 dist-test-slave-hwlj krb5kdc[15223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:53:53.960362 15842 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 20:53:53.960942 15842 webserver.cc:460] Webserver started at http://127.0.98.1:36875/ using document root <none> and password file <none>
I20260926 20:53:53.961225 15842 fs_manager.cc:359] Metadata directory not provided
I20260926 20:53:53.961310 15842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:53:53.963227 15842 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260926 20:53:53.964372 15877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:53.964680 15842 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:53:53.964780 15842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-0/wal
uuid: "9a0a759577424f46b7341380af800802"
format_stamp: "Formatted at 2026-09-26 20:53:33 on dist-test-slave-hwlj"
server_key: "ee6e85469a0994cf33e261b7008e0ed3"
server_key_iv: "e5e60cc76a891fff44870d8472b99adc"
server_key_version: "encryptionkey@0"
I20260926 20:53:53.965247 15842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:53:53.989907 15842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:53:53.992758 15842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:53:53.992923 15842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:53:53.993247 15842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:53:53.993791 15884 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 20:53:53.995291 15842 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 20:53:53.995358 15842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 20:53:53.995398 15842 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 20:53:53.996325 15842 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 20:53:53.996415 15842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.001s
I20260926 20:53:53.996618 15884 tablet_bootstrap.cc:492] T 0c74fe380fdf407a8fd25cef6e5382c2 P 9a0a759577424f46b7341380af800802: Bootstrap starting.
I20260926 20:53:54.003799 15842 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41173
I20260926 20:53:54.003917 15991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41173 every 8 connection(s)
I20260926 20:53:54.004428 15842 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:53:54.007850 15884 log.cc:824] T 0c74fe380fdf407a8fd25cef6e5382c2 P 9a0a759577424f46b7341380af800802: Log is configured to *not* fsync() on all Append() calls
I20260926 20:53:54.010408 15884 tablet_bootstrap.cc:492] T 0c74fe380fdf407a8fd25cef6e5382c2 P 9a0a759577424f46b7341380af800802: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:53:54.010730 15884 tablet_bootstrap.cc:492] T 0c74fe380fdf407a8fd25cef6e5382c2 P 9a0a759577424f46b7341380af800802: Bootstrap complete.
I20260926 20:53:54.011842 15884 ts_tablet_manager.cc:1397] T 0c74fe380fdf407a8fd25cef6e5382c2 P 9a0a759577424f46b7341380af800802: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.008s
I20260926 20:53:54.012574 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 15842
I20260926 20:53:54.012928 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41677
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43565
--webserver_port=46305
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:53:54.013996 15884 raft_consensus.cc:348] T 0c74fe380fdf407a8fd25cef6e5382c2 P 9a0a759577424f46b7341380af800802 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a0a759577424f46b7341380af800802" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41173 } } peers { permanent_uuid: "1788a70030a64ae7b5059ac6a1700692" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40147 } } peers { permanent_uuid: "fb366842fb274b3fa8b268cece62eab0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41677 } }
Sep 26 20:53:54 dist-test-slave-hwlj krb5kdc[15223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:53:54.014391 15884 raft_consensus.cc:729] T 0c74fe380fdf407a8fd25cef6e5382c2 P 9a0a759577424f46b7341380af800802 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a0a759577424f46b7341380af800802, State: Initialized, Role: FOLLOWER
I20260926 20:53:54.014575 15884 consensus_queue.cc:260] T 0c74fe380fdf407a8fd25cef6e5382c2 P 9a0a759577424f46b7341380af800802 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a0a759577424f46b7341380af800802" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41173 } } peers { permanent_uuid: "1788a70030a64ae7b5059ac6a1700692" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40147 } } peers { permanent_uuid: "fb366842fb274b3fa8b268cece62eab0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41677 } }
I20260926 20:53:54.014885 15884 ts_tablet_manager.cc:1428] T 0c74fe380fdf407a8fd25cef6e5382c2 P 9a0a759577424f46b7341380af800802: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260926 20:53:54.019227 15995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:54.005444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35569 (local address 127.0.98.62:43565)
0926 20:53:54.007149 (+ 1705us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:54.007162 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:54.007210 (+ 48us) server_negotiation.cc:407] Connection header received
0926 20:53:54.007360 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:54.007368 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:54.007435 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:54.007591 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:54.008493 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:54.009896 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:54.011384 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:54.012101 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:54.012209 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:54.015052 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:54.015057 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:54.015057 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:54.015069 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:54.017253 (+ 2184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:54.017732 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:54.017735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:54.017735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:54.017845 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:54.018141 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:54.018142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:54.018142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:54.018424 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:54.018479 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:54.018852 (+ 373us) server_negotiation.cc:299] Negotiation successful
0926 20:53:54.019022 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1641,"thread_start_us":89,"threads_started":1}
I20260926 20:53:54.019819 15994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:54.005162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.1:35569)
0926 20:53:54.005370 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:54.005434 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:54.006172 (+ 738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:54.007654 (+ 1482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:54.007659 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:54.008084 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:54.008353 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:54.008360 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:54.010053 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:54.010054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:54.011288 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:54.011291 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:54.012215 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:54.012217 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:54.012363 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:54.013051 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:54.013063 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:54.014907 (+ 1844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:53:54.017349 (+ 2442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:54.017356 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:54.017358 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:54.017635 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:54.017920 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:54.017920 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:54.017921 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:54.018062 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:54.018530 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:53:54.018536 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:53:54.018680 (+ 144us) client_negotiation.cc:769] Sending connection context
0926 20:53:54.019391 (+ 711us) client_negotiation.cc:240] Negotiation successful
0926 20:53:54.019573 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":100,"threads_started":1}
I20260926 20:53:54.020661 15992 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43565
I20260926 20:53:54.020773 15992 heartbeater.cc:464] Registering TS with master...
I20260926 20:53:54.020974 15992 heartbeater.cc:511] Master 127.0.98.62:43565 requested a full tablet report, sending...
I20260926 20:53:54.021816 15804 ts_manager.cc:194] Registered new tserver with Master: 9a0a759577424f46b7341380af800802 (127.0.98.1:41173)
I20260926 20:53:54.023139 15804 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35569
I20260926 20:53:54.024312 15992 heartbeater.cc:503] Master 127.0.98.62:43565 was elected leader, sending a full tablet report...
I20260926 20:53:54.028465 16000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:54.022745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41694 (local address 127.0.98.1:41173)
0926 20:53:54.022912 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:54.022917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:54.023946 (+ 1029us) server_negotiation.cc:407] Connection header received
0926 20:53:54.024085 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:54.024087 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:54.024527 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:54.024606 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:53:54.024870 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:54.025929 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:54.027444 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:54.028091 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:54.028164 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:54.028214 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:54.028370 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 20:53:54.028407 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":93,"threads_started":1}
I20260926 20:53:54.028470 15999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:54.022505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41173 (local address 127.0.0.1:41694)
0926 20:53:54.023852 (+ 1347us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:54.023889 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:54.023981 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:54.024675 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:54.024679 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:54.024681 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:53:54.024745 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:54.024746 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:54.026013 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:54.026014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:54.027360 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:54.027362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:54.028173 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:54.028174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:54.028256 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:54.028270 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 20:53:54.028345 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 20:53:54.028394 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1318,"thread_start_us":88,"threads_started":1}
I20260926 20:53:54.029280 15926 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c74fe380fdf407a8fd25cef6e5382c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:53:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41694
I20260926 20:53:54.029862 16001 tablet_replica.cc:330] stopping tablet replica
I20260926 20:53:54.029969 16001 raft_consensus.cc:2229] T 0c74fe380fdf407a8fd25cef6e5382c2 P 9a0a759577424f46b7341380af800802 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:53:54.030035 16001 raft_consensus.cc:2258] T 0c74fe380fdf407a8fd25cef6e5382c2 P 9a0a759577424f46b7341380af800802 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:53:54.030371 16001 ts_tablet_manager.cc:1905] T 0c74fe380fdf407a8fd25cef6e5382c2 P 9a0a759577424f46b7341380af800802: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:53:54.032897 16001 ts_tablet_manager.cc:1918] T 0c74fe380fdf407a8fd25cef6e5382c2 P 9a0a759577424f46b7341380af800802: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:53:54.032974 16001 log.cc:1196] T 0c74fe380fdf407a8fd25cef6e5382c2 P 9a0a759577424f46b7341380af800802: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-0/wal/wals/0c74fe380fdf407a8fd25cef6e5382c2
I20260926 20:53:54.033214 16001 ts_tablet_manager.cc:1939] T 0c74fe380fdf407a8fd25cef6e5382c2 P 9a0a759577424f46b7341380af800802: Deleting consensus metadata
I20260926 20:53:54.033725 15791 catalog_manager.cc:4841] TS 9a0a759577424f46b7341380af800802 (127.0.98.1:41173): tablet 0c74fe380fdf407a8fd25cef6e5382c2 (table test-table [id=24ff8b191b684491aa49d6171977f194]) successfully deleted
W20260926 20:53:54.099255 15997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:53:54.099460 15997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:53:54.099484 15997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:53:54.100610 15997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:53:54.100647 15997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:53:54.100673 15997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:53:54.102056 15997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41677
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46305
--tserver_master_addrs=127.0.98.62:43565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:53:54.102243 15997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:53:54.102417 15997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:53:54.107402 16007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:54.107613 15997 server_base.cc:1034] running on GCE node
W20260926 20:53:54.107434 16009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:54.107419 16006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:54.108018 15997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:53:54.108289 15997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:53:54.109467 15997 hybrid_clock.cc:648] HybridClock initialized: now 1790456034109453 us; error 28 us; skew 500 ppm
Sep 26 20:53:54 dist-test-slave-hwlj krb5kdc[15223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:53:54.112149 15997 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 20:53:54.112665 15997 webserver.cc:460] Webserver started at http://127.0.98.2:46305/ using document root <none> and password file <none>
I20260926 20:53:54.112964 15997 fs_manager.cc:359] Metadata directory not provided
I20260926 20:53:54.113027 15997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:53:54.114538 15997 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:53:54.115135 16016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:54.115309 15997 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:53:54.115392 15997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-1/wal
uuid: "fb366842fb274b3fa8b268cece62eab0"
format_stamp: "Formatted at 2026-09-26 20:53:33 on dist-test-slave-hwlj"
server_key: "d3dde8a914d4b2b7e8e057f788e4ab75"
server_key_iv: "f98a5af4f03254e4fe9e351b774e53f4"
server_key_version: "encryptionkey@0"
I20260926 20:53:54.115738 15997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:53:54.148753 15997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:53:54.151320 15997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:53:54.151438 15997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:53:54.151633 15997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:53:54.151984 15997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:53:54.152021 15997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:54.152045 15997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:53:54.152097 15997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:54.157961 15997 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41677
I20260926 20:53:54.158066 16129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41677 every 8 connection(s)
I20260926 20:53:54.158391 15997 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
I20260926 20:53:54.159544 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 15997
I20260926 20:53:54.159888 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40147
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43565
--webserver_port=41589
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:53:54 dist-test-slave-hwlj krb5kdc[15223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:53:54.169066 15995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:54.159171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34091 (local address 127.0.98.62:43565)
0926 20:53:54.159242 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:54.159244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:54.159499 (+ 255us) server_negotiation.cc:407] Connection header received
0926 20:53:54.160168 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:54.160170 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:54.160183 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:54.160261 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:54.161217 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:54.161895 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:54.163132 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:54.163426 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:54.163510 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:54.166064 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:54.166066 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:54.166066 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:54.166074 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:54.167524 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:54.168048 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:54.168050 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:54.168050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:54.168117 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:54.168387 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:54.168388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:54.168388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:54.168508 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:54.168585 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:54.168869 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 20:53:54.168961 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 20:53:54.169255 16132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:54.159163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.2:34091)
0926 20:53:54.159347 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:54.159396 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:54.160060 (+ 664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:54.160317 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:54.160321 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:54.160881 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:54.161081 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:54.161086 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:54.162000 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:54.162001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:54.163030 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:54.163034 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:54.163522 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:54.163523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:54.163657 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:54.164247 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:54.164255 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:54.165951 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:53:54.167634 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:54.167643 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:54.167645 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:54.167958 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:54.168191 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:54.168192 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:54.168192 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:54.168313 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:54.168577 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:53:54.168584 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:53:54.168734 (+ 150us) client_negotiation.cc:769] Sending connection context
0926 20:53:54.168838 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 20:53:54.169000 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":100,"threads_started":1}
I20260926 20:53:54.169871 16130 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43565
I20260926 20:53:54.169987 16130 heartbeater.cc:464] Registering TS with master...
I20260926 20:53:54.170178 16130 heartbeater.cc:511] Master 127.0.98.62:43565 requested a full tablet report, sending...
I20260926 20:53:54.170575 15804 ts_manager.cc:194] Registered new tserver with Master: fb366842fb274b3fa8b268cece62eab0 (127.0.98.2:41677)
I20260926 20:53:54.170961 15804 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34091
W20260926 20:53:54.243582 16133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:53:54.243897 16133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:53:54.243944 16133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:53:54.245193 16133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:53:54.245270 16133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:53:54.245316 16133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:53:54.246845 16133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40147
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41589
--tserver_master_addrs=127.0.98.62:43565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:53:54.247133 16133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:53:54.247376 16133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:53:54.252491 16139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:54.252503 16138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:54.252491 16141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:54.252868 16133 server_base.cc:1034] running on GCE node
I20260926 20:53:54.253116 16133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:53:54.253324 16133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:53:54.254462 16133 hybrid_clock.cc:648] HybridClock initialized: now 1790456034254449 us; error 28 us; skew 500 ppm
Sep 26 20:53:54 dist-test-slave-hwlj krb5kdc[15223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:53:54.257032 16133 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 20:53:54.257517 16133 webserver.cc:460] Webserver started at http://127.0.98.3:41589/ using document root <none> and password file <none>
I20260926 20:53:54.257723 16133 fs_manager.cc:359] Metadata directory not provided
I20260926 20:53:54.257776 16133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:53:54.259083 16133 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:53:54.259677 16148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:54.259881 16133 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:53:54.259989 16133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-2/wal
uuid: "1788a70030a64ae7b5059ac6a1700692"
format_stamp: "Formatted at 2026-09-26 20:53:33 on dist-test-slave-hwlj"
server_key: "dfbb201cea7e108042799806bed68557"
server_key_iv: "894f56d5750433eda96deba6a9e04248"
server_key_version: "encryptionkey@0"
I20260926 20:53:54.260345 16133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:53:54.271188 16133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:53:54.273162 16133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:53:54.273335 16133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:53:54.273574 16133 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:53:54.273938 16133 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:53:54.273994 16133 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:54.274036 16133 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:53:54.274065 16133 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:54.279947 16133 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40147
I20260926 20:53:54.279996 16261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40147 every 8 connection(s)
I20260926 20:53:54.280334 16133 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
I20260926 20:53:54.285236 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 16133
Sep 26 20:53:54 dist-test-slave-hwlj krb5kdc[15223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:53:54.290508 15995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:54.281043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42913 (local address 127.0.98.62:43565)
0926 20:53:54.281090 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:54.281092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:54.281307 (+ 215us) server_negotiation.cc:407] Connection header received
0926 20:53:54.281870 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:54.281871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:54.281882 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:54.281934 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:54.282874 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:54.283397 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:54.284526 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:54.284724 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:54.284780 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:54.287412 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:54.287413 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:54.287413 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:54.287428 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:54.289026 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:54.289500 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:54.289500 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:54.289501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:54.289551 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:54.289849 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:54.289850 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:54.289850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:54.289991 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:54.290093 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:54.290338 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 20:53:54.290418 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 20:53:54.290696 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:54.281035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.3:42913)
0926 20:53:54.281181 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:54.281220 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:54.281779 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:54.281990 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:54.281994 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:54.282578 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:54.282770 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:54.282774 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:54.283484 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:54.283485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:54.284454 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:54.284457 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:54.284819 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:54.284820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:54.284933 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:54.285569 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:54.285582 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:54.287300 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:53:54.289105 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:54.289112 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:54.289114 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:54.289414 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:54.289638 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:54.289640 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:54.289640 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:54.289780 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:54.290068 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:53:54.290073 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:53:54.290190 (+ 117us) client_negotiation.cc:769] Sending connection context
0926 20:53:54.290299 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 20:53:54.290456 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":77,"threads_started":1}
I20260926 20:53:54.291265 16262 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43565
I20260926 20:53:54.291378 16262 heartbeater.cc:464] Registering TS with master...
I20260926 20:53:54.291589 16262 heartbeater.cc:511] Master 127.0.98.62:43565 requested a full tablet report, sending...
I20260926 20:53:54.292047 15804 ts_manager.cc:194] Registered new tserver with Master: 1788a70030a64ae7b5059ac6a1700692 (127.0.98.3:40147)
I20260926 20:53:54.292532 15804 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42913
I20260926 20:53:54.292748 16266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:54.285676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42230 (local address 127.0.98.62:43565)
0926 20:53:54.285777 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:54.285779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:54.285873 (+ 94us) server_negotiation.cc:407] Connection header received
0926 20:53:54.285938 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:54.285939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:54.285947 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:54.286003 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:54.286720 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:54.287300 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:54.288143 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:54.288448 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:54.288525 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:54.289942 (+ 1417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:54.289944 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:54.289944 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:54.289950 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:54.291528 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:54.291969 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:54.291970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:54.291970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:54.292034 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:54.292250 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:54.292251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:54.292252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:54.292366 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:54.292527 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:54.292565 (+ 38us) server_negotiation.cc:299] Negotiation successful
0926 20:53:54.292644 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":52,"threads_started":1}
I20260926 20:53:54.293490 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:53:54.296134 16266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:54.293846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42238 (local address 127.0.98.62:43565)
0926 20:53:54.293899 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:54.293901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:54.294026 (+ 125us) server_negotiation.cc:407] Connection header received
0926 20:53:54.294073 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:54.294074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:54.294101 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:54.294163 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:53:54.294257 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:54.294698 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:54.295464 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:54.295645 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:54.295693 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:54.295926 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:53:54.295965 (+ 39us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 20:53:54.296091 (+ 126us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42238: token expired
Metrics: {"server-negotiator.queue_time_us":27}
W20260926 20:53:54.296197 16266 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42238: token expired
W20260926 20:53:54.296311 15731 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:43565)
I20260926 20:53:54.302408 16266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:54.296604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42250 (local address 127.0.98.62:43565)
0926 20:53:54.296662 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:54.296664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:54.296672 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:53:54.296742 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:54.296743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:54.296752 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:54.296799 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:54.297371 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:54.297822 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:54.298529 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:54.298724 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:54.298807 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:54.299988 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:54.299990 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:54.299990 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:54.299997 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:54.301234 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:54.301602 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:54.301603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:54.301603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:54.301663 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:54.301862 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:54.301863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:54.301863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:54.302003 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:54.302132 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:54.302235 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 20:53:54.302313 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 20:53:54.303582 15804 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:53:54.305217 15804 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:53:54.307308 15926 tablet_service.cc:1467] Processing CreateTablet for tablet d1c08f190206437b994a91585c71ad78 (DEFAULT_TABLE table=test-table [id=c22ac9799c6e46f596d148c7c23a44c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:53:54.307512 15926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1c08f190206437b994a91585c71ad78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:53:54.308818 15884 tablet_bootstrap.cc:492] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802: Bootstrap starting.
I20260926 20:53:54.309168 15884 tablet_bootstrap.cc:654] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802: Neither blocks nor log segments found. Creating new log.
I20260926 20:53:54.309696 15884 tablet_bootstrap.cc:492] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802: No bootstrap required, opened a new log
I20260926 20:53:54.309759 15884 ts_tablet_manager.cc:1397] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260926 20:53:54.309875 15884 raft_consensus.cc:348] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb366842fb274b3fa8b268cece62eab0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41677 } } peers { permanent_uuid: "1788a70030a64ae7b5059ac6a1700692" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40147 } } peers { permanent_uuid: "9a0a759577424f46b7341380af800802" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41173 } }
I20260926 20:53:54.309923 15884 raft_consensus.cc:374] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:53:54.309950 15884 raft_consensus.cc:729] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a0a759577424f46b7341380af800802, State: Initialized, Role: FOLLOWER
I20260926 20:53:54.310006 15884 consensus_queue.cc:260] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb366842fb274b3fa8b268cece62eab0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41677 } } peers { permanent_uuid: "1788a70030a64ae7b5059ac6a1700692" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40147 } } peers { permanent_uuid: "9a0a759577424f46b7341380af800802" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41173 } }
I20260926 20:53:54.310105 15884 ts_tablet_manager.cc:1428] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:54.311271 15999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:54.307062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40147 (local address 127.0.0.1:54184)
0926 20:53:54.307110 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:54.307121 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:54.307159 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:54.307629 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:54.307630 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:54.307632 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:53:54.307673 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:54.307675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:54.308973 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:54.308974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:54.310100 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:54.310103 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:54.311057 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:54.311059 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:54.311123 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:54.311126 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:53:54.311156 (+ 30us) client_negotiation.cc:240] Negotiation successful
0926 20:53:54.311200 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 20:53:54.311390 16270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:54.307428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54010 (local address 127.0.98.2:41677)
0926 20:53:54.307600 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:54.307604 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:54.307614 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:53:54.307653 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:54.307654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:54.307671 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:54.307916 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:53:54.308000 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:54.309044 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:54.310362 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:54.311011 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:54.311116 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:54.311184 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:54.311256 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 20:53:54.311298 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":94,"threads_started":1}
I20260926 20:53:54.311398 16268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:54.307076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41677 (local address 127.0.0.1:54010)
0926 20:53:54.307227 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:54.307235 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:54.307278 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:54.307806 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:54.307807 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:54.307809 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:53:54.307847 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:54.307848 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:54.309138 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:54.309139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:54.310205 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:54.310207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:54.311118 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:54.311119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:54.311190 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:54.311194 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:53:54.311243 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 20:53:54.311274 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":92,"threads_started":1}
I20260926 20:53:54.311700 16269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:54.307233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54184 (local address 127.0.98.3:40147)
0926 20:53:54.307447 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:54.307452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:54.307463 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:53:54.307502 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:54.307504 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:54.307521 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:54.307634 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:53:54.307729 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:54.308876 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:54.310204 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:54.310953 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:54.311452 (+ 499us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:54.311528 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:54.311556 (+ 28us) server_negotiation.cc:299] Negotiation successful
0926 20:53:54.311614 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":94,"threads_started":1}
I20260926 20:53:54.312198 16064 tablet_service.cc:1467] Processing CreateTablet for tablet d1c08f190206437b994a91585c71ad78 (DEFAULT_TABLE table=test-table [id=c22ac9799c6e46f596d148c7c23a44c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:53:54.312325 16196 tablet_service.cc:1467] Processing CreateTablet for tablet d1c08f190206437b994a91585c71ad78 (DEFAULT_TABLE table=test-table [id=c22ac9799c6e46f596d148c7c23a44c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:53:54.312546 16064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1c08f190206437b994a91585c71ad78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:53:54.312685 16196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1c08f190206437b994a91585c71ad78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:53:54.315455 16273 tablet_bootstrap.cc:492] T d1c08f190206437b994a91585c71ad78 P fb366842fb274b3fa8b268cece62eab0: Bootstrap starting.
I20260926 20:53:54.315455 16272 tablet_bootstrap.cc:492] T d1c08f190206437b994a91585c71ad78 P 1788a70030a64ae7b5059ac6a1700692: Bootstrap starting.
I20260926 20:53:54.316206 16272 tablet_bootstrap.cc:654] T d1c08f190206437b994a91585c71ad78 P 1788a70030a64ae7b5059ac6a1700692: Neither blocks nor log segments found. Creating new log.
I20260926 20:53:54.316278 16273 tablet_bootstrap.cc:654] T d1c08f190206437b994a91585c71ad78 P fb366842fb274b3fa8b268cece62eab0: Neither blocks nor log segments found. Creating new log.
I20260926 20:53:54.316720 16272 log.cc:824] T d1c08f190206437b994a91585c71ad78 P 1788a70030a64ae7b5059ac6a1700692: Log is configured to *not* fsync() on all Append() calls
I20260926 20:53:54.316757 16273 log.cc:824] T d1c08f190206437b994a91585c71ad78 P fb366842fb274b3fa8b268cece62eab0: Log is configured to *not* fsync() on all Append() calls
I20260926 20:53:54.317508 16273 tablet_bootstrap.cc:492] T d1c08f190206437b994a91585c71ad78 P fb366842fb274b3fa8b268cece62eab0: No bootstrap required, opened a new log
I20260926 20:53:54.317515 16272 tablet_bootstrap.cc:492] T d1c08f190206437b994a91585c71ad78 P 1788a70030a64ae7b5059ac6a1700692: No bootstrap required, opened a new log
I20260926 20:53:54.317584 16273 ts_tablet_manager.cc:1397] T d1c08f190206437b994a91585c71ad78 P fb366842fb274b3fa8b268cece62eab0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 20:53:54.317588 16272 ts_tablet_manager.cc:1397] T d1c08f190206437b994a91585c71ad78 P 1788a70030a64ae7b5059ac6a1700692: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:53:54.319523 16272 raft_consensus.cc:348] T d1c08f190206437b994a91585c71ad78 P 1788a70030a64ae7b5059ac6a1700692 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb366842fb274b3fa8b268cece62eab0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41677 } } peers { permanent_uuid: "1788a70030a64ae7b5059ac6a1700692" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40147 } } peers { permanent_uuid: "9a0a759577424f46b7341380af800802" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41173 } }
I20260926 20:53:54.319540 16273 raft_consensus.cc:348] T d1c08f190206437b994a91585c71ad78 P fb366842fb274b3fa8b268cece62eab0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb366842fb274b3fa8b268cece62eab0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41677 } } peers { permanent_uuid: "1788a70030a64ae7b5059ac6a1700692" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40147 } } peers { permanent_uuid: "9a0a759577424f46b7341380af800802" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41173 } }
I20260926 20:53:54.319718 16273 raft_consensus.cc:374] T d1c08f190206437b994a91585c71ad78 P fb366842fb274b3fa8b268cece62eab0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:53:54.319718 16272 raft_consensus.cc:374] T d1c08f190206437b994a91585c71ad78 P 1788a70030a64ae7b5059ac6a1700692 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:53:54.319782 16272 raft_consensus.cc:729] T d1c08f190206437b994a91585c71ad78 P 1788a70030a64ae7b5059ac6a1700692 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1788a70030a64ae7b5059ac6a1700692, State: Initialized, Role: FOLLOWER
I20260926 20:53:54.319782 16273 raft_consensus.cc:729] T d1c08f190206437b994a91585c71ad78 P fb366842fb274b3fa8b268cece62eab0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb366842fb274b3fa8b268cece62eab0, State: Initialized, Role: FOLLOWER
I20260926 20:53:54.319909 16272 consensus_queue.cc:260] T d1c08f190206437b994a91585c71ad78 P 1788a70030a64ae7b5059ac6a1700692 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb366842fb274b3fa8b268cece62eab0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41677 } } peers { permanent_uuid: "1788a70030a64ae7b5059ac6a1700692" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40147 } } peers { permanent_uuid: "9a0a759577424f46b7341380af800802" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41173 } }
I20260926 20:53:54.319909 16273 consensus_queue.cc:260] T d1c08f190206437b994a91585c71ad78 P fb366842fb274b3fa8b268cece62eab0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb366842fb274b3fa8b268cece62eab0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41677 } } peers { permanent_uuid: "1788a70030a64ae7b5059ac6a1700692" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40147 } } peers { permanent_uuid: "9a0a759577424f46b7341380af800802" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41173 } }
I20260926 20:53:54.320211 16273 ts_tablet_manager.cc:1428] T d1c08f190206437b994a91585c71ad78 P fb366842fb274b3fa8b268cece62eab0: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260926 20:53:54.320211 16272 ts_tablet_manager.cc:1428] T d1c08f190206437b994a91585c71ad78 P 1788a70030a64ae7b5059ac6a1700692: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:53:54.320232 16130 heartbeater.cc:503] Master 127.0.98.62:43565 was elected leader, sending a full tablet report...
I20260926 20:53:54.320326 16262 heartbeater.cc:503] Master 127.0.98.62:43565 was elected leader, sending a full tablet report...
W20260926 20:53:54.409009 16131 tablet.cc:2369] T d1c08f190206437b994a91585c71ad78 P fb366842fb274b3fa8b268cece62eab0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:53:54.425611 16277 raft_consensus.cc:482] T d1c08f190206437b994a91585c71ad78 P 1788a70030a64ae7b5059ac6a1700692 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:53:54.425748 16277 raft_consensus.cc:504] T d1c08f190206437b994a91585c71ad78 P 1788a70030a64ae7b5059ac6a1700692 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb366842fb274b3fa8b268cece62eab0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41677 } } peers { permanent_uuid: "1788a70030a64ae7b5059ac6a1700692" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40147 } } peers { permanent_uuid: "9a0a759577424f46b7341380af800802" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41173 } }
I20260926 20:53:54.426198 16277 leader_election.cc:288] T d1c08f190206437b994a91585c71ad78 P 1788a70030a64ae7b5059ac6a1700692 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb366842fb274b3fa8b268cece62eab0 (127.0.98.2:41677), 9a0a759577424f46b7341380af800802 (127.0.98.1:41173)
I20260926 20:53:54.429229 16000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:54.426470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50721 (local address 127.0.98.1:41173)
0926 20:53:54.426526 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:54.426528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:54.426538 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:53:54.426568 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:54.426569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:54.426581 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:54.426640 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:53:54.426966 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:54.427480 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:54.428485 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:54.428857 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:54.428951 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:54.428963 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:54.429126 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 20:53:54.429167 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 20:53:54.429229 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:54.426386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41173 (local address 127.0.98.3:50721)
0926 20:53:54.426437 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:54.426449 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:54.426492 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:54.426661 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:54.426662 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:54.426664 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:53:54.426762 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:54.426764 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:54.427581 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:54.427583 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:54.428377 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:54.428380 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:54.428952 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:54.428953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:54.429036 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:54.429044 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 20:53:54.429109 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 20:53:54.429152 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":3}
I20260926 20:53:54.429311 16270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:54.426476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47343 (local address 127.0.98.2:41677)
0926 20:53:54.426519 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:54.426521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:54.426839 (+ 318us) server_negotiation.cc:407] Connection header received
0926 20:53:54.426871 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:54.426872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:54.426882 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:54.426924 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:53:54.427064 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:54.427537 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:54.428564 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:54.428943 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:54.429025 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:54.429035 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:54.429197 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 20:53:54.429233 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 20:53:54.429549 16278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:54.426394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41677 (local address 127.0.98.3:47343)
0926 20:53:54.426751 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:54.426761 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:54.426825 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:54.426944 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:54.426945 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:54.426946 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:53:54.426992 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:54.426992 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:54.427642 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:54.427643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:54.428459 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:54.428461 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:54.429028 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:54.429029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:54.429104 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:54.429108 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:53:54.429180 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 20:53:54.429220 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":153,"thread_start_us":120,"threads_started":1}
I20260926 20:53:54.429569 15946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1c08f190206437b994a91585c71ad78" candidate_uuid: "1788a70030a64ae7b5059ac6a1700692" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a0a759577424f46b7341380af800802" is_pre_election: true
I20260926 20:53:54.429726 15946 raft_consensus.cc:2454] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1788a70030a64ae7b5059ac6a1700692 in term 0.
I20260926 20:53:54.430014 16152 leader_election.cc:302] T d1c08f190206437b994a91585c71ad78 P 1788a70030a64ae7b5059ac6a1700692 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1788a70030a64ae7b5059ac6a1700692, 9a0a759577424f46b7341380af800802; no voters:
I20260926 20:53:54.430147 16084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1c08f190206437b994a91585c71ad78" candidate_uuid: "1788a70030a64ae7b5059ac6a1700692" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb366842fb274b3fa8b268cece62eab0" is_pre_election: true
I20260926 20:53:54.430177 16277 raft_consensus.cc:2781] T d1c08f190206437b994a91585c71ad78 P 1788a70030a64ae7b5059ac6a1700692 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:53:54.430261 16277 raft_consensus.cc:482] T d1c08f190206437b994a91585c71ad78 P 1788a70030a64ae7b5059ac6a1700692 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:53:54.430284 16084 raft_consensus.cc:2454] T d1c08f190206437b994a91585c71ad78 P fb366842fb274b3fa8b268cece62eab0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1788a70030a64ae7b5059ac6a1700692 in term 0.
I20260926 20:53:54.430299 16277 raft_consensus.cc:3037] T d1c08f190206437b994a91585c71ad78 P 1788a70030a64ae7b5059ac6a1700692 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:53:54.431054 16277 raft_consensus.cc:504] T d1c08f190206437b994a91585c71ad78 P 1788a70030a64ae7b5059ac6a1700692 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb366842fb274b3fa8b268cece62eab0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41677 } } peers { permanent_uuid: "1788a70030a64ae7b5059ac6a1700692" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40147 } } peers { permanent_uuid: "9a0a759577424f46b7341380af800802" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41173 } }
I20260926 20:53:54.431212 16277 leader_election.cc:288] T d1c08f190206437b994a91585c71ad78 P 1788a70030a64ae7b5059ac6a1700692 [CANDIDATE]: Term 1 election: Requested vote from peers fb366842fb274b3fa8b268cece62eab0 (127.0.98.2:41677), 9a0a759577424f46b7341380af800802 (127.0.98.1:41173)
I20260926 20:53:54.431376 16084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1c08f190206437b994a91585c71ad78" candidate_uuid: "1788a70030a64ae7b5059ac6a1700692" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb366842fb274b3fa8b268cece62eab0"
I20260926 20:53:54.431473 16084 raft_consensus.cc:3037] T d1c08f190206437b994a91585c71ad78 P fb366842fb274b3fa8b268cece62eab0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:53:54.431393 15946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1c08f190206437b994a91585c71ad78" candidate_uuid: "1788a70030a64ae7b5059ac6a1700692" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a0a759577424f46b7341380af800802"
I20260926 20:53:54.431588 15946 raft_consensus.cc:3037] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:53:54.432260 16084 raft_consensus.cc:2454] T d1c08f190206437b994a91585c71ad78 P fb366842fb274b3fa8b268cece62eab0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1788a70030a64ae7b5059ac6a1700692 in term 1.
I20260926 20:53:54.432435 16151 leader_election.cc:302] T d1c08f190206437b994a91585c71ad78 P 1788a70030a64ae7b5059ac6a1700692 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1788a70030a64ae7b5059ac6a1700692, fb366842fb274b3fa8b268cece62eab0; no voters:
I20260926 20:53:54.432518 16277 raft_consensus.cc:2781] T d1c08f190206437b994a91585c71ad78 P 1788a70030a64ae7b5059ac6a1700692 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:53:54.432667 16277 raft_consensus.cc:686] T d1c08f190206437b994a91585c71ad78 P 1788a70030a64ae7b5059ac6a1700692 [term 1 LEADER]: Becoming Leader. State: Replica: 1788a70030a64ae7b5059ac6a1700692, State: Running, Role: LEADER
I20260926 20:53:54.432770 16277 consensus_queue.cc:237] T d1c08f190206437b994a91585c71ad78 P 1788a70030a64ae7b5059ac6a1700692 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb366842fb274b3fa8b268cece62eab0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41677 } } peers { permanent_uuid: "1788a70030a64ae7b5059ac6a1700692" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40147 } } peers { permanent_uuid: "9a0a759577424f46b7341380af800802" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41173 } }
I20260926 20:53:54.432931 15946 raft_consensus.cc:2454] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1788a70030a64ae7b5059ac6a1700692 in term 1.
I20260926 20:53:54.433616 15804 catalog_manager.cc:5495] T d1c08f190206437b994a91585c71ad78 P 1788a70030a64ae7b5059ac6a1700692 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1788a70030a64ae7b5059ac6a1700692 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "1788a70030a64ae7b5059ac6a1700692" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb366842fb274b3fa8b268cece62eab0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1788a70030a64ae7b5059ac6a1700692" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a0a759577424f46b7341380af800802" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41173 } health_report { overall_health: UNKNOWN } } }
I20260926 20:53:54.459263 16269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:54.456970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54200 (local address 127.0.98.3:40147)
0926 20:53:54.457020 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:54.457026 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:54.457057 (+ 31us) server_negotiation.cc:407] Connection header received
0926 20:53:54.457091 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:54.457091 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:54.457111 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:54.457169 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:53:54.457317 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:54.457839 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:54.458558 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:54.458759 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:54.458818 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:54.458956 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:53:54.459057 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:53:54.459117 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:54.459158 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 20:53:54.459175 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 20:53:54.461316 15946 raft_consensus.cc:1261] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802 [term 1 FOLLOWER]: Refusing update from remote peer 1788a70030a64ae7b5059ac6a1700692: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:53:54.461336 16084 raft_consensus.cc:1261] T d1c08f190206437b994a91585c71ad78 P fb366842fb274b3fa8b268cece62eab0 [term 1 FOLLOWER]: Refusing update from remote peer 1788a70030a64ae7b5059ac6a1700692: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:53:54.461741 16279 consensus_queue.cc:1035] T d1c08f190206437b994a91585c71ad78 P 1788a70030a64ae7b5059ac6a1700692 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0a759577424f46b7341380af800802" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:53:54.461920 16277 consensus_queue.cc:1035] T d1c08f190206437b994a91585c71ad78 P 1788a70030a64ae7b5059ac6a1700692 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb366842fb274b3fa8b268cece62eab0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:53:54.464514 16282 mvcc.cc:205] Tried to move back new op lower bound from 7333707917150580736 to 7333707917037293568. Current Snapshot: MvccSnapshot[applied={T|T < 7333707917150580736}]
I20260926 20:53:54.464843 16283 mvcc.cc:205] Tried to move back new op lower bound from 7333707917150580736 to 7333707917037293568. Current Snapshot: MvccSnapshot[applied={T|T < 7333707917150580736}]
I20260926 20:53:54.465796 16285 mvcc.cc:205] Tried to move back new op lower bound from 7333707917150580736 to 7333707917037293568. Current Snapshot: MvccSnapshot[applied={T|T < 7333707917150580736 or (T in {7333707917150580736})}]
I20260926 20:53:54.467267 15804 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:53:54.467368 15804 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:53:54.468077 15804 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe: Sending DeleteTablet for 3 replicas of tablet d1c08f190206437b994a91585c71ad78
I20260926 20:53:54.468369 16196 tablet_service.cc:1514] Processing DeleteTablet for tablet d1c08f190206437b994a91585c71ad78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:53:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54184
I20260926 20:53:54.468474 16064 tablet_service.cc:1514] Processing DeleteTablet for tablet d1c08f190206437b994a91585c71ad78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:53:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54010
I20260926 20:53:54.468607 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 15842
I20260926 20:53:54.468704 16292 tablet_replica.cc:330] T d1c08f190206437b994a91585c71ad78 P fb366842fb274b3fa8b268cece62eab0: stopping tablet replica
I20260926 20:53:54.468818 16292 raft_consensus.cc:2229] T d1c08f190206437b994a91585c71ad78 P fb366842fb274b3fa8b268cece62eab0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:53:54.468930 16292 raft_consensus.cc:2258] T d1c08f190206437b994a91585c71ad78 P fb366842fb274b3fa8b268cece62eab0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:53:54.469604 16291 tablet_replica.cc:330] T d1c08f190206437b994a91585c71ad78 P 1788a70030a64ae7b5059ac6a1700692: stopping tablet replica
I20260926 20:53:54.469708 16291 raft_consensus.cc:2229] T d1c08f190206437b994a91585c71ad78 P 1788a70030a64ae7b5059ac6a1700692 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:53:54.469832 16291 raft_consensus.cc:2258] T d1c08f190206437b994a91585c71ad78 P 1788a70030a64ae7b5059ac6a1700692 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:53:54.470268 16291 ts_tablet_manager.cc:1905] T d1c08f190206437b994a91585c71ad78 P 1788a70030a64ae7b5059ac6a1700692: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:53:54.470741 16292 ts_tablet_manager.cc:1905] T d1c08f190206437b994a91585c71ad78 P fb366842fb274b3fa8b268cece62eab0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:53:54.472591 16291 ts_tablet_manager.cc:1918] T d1c08f190206437b994a91585c71ad78 P 1788a70030a64ae7b5059ac6a1700692: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:53:54.472630 16292 ts_tablet_manager.cc:1918] T d1c08f190206437b994a91585c71ad78 P fb366842fb274b3fa8b268cece62eab0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:53:54.472674 16291 log.cc:1196] T d1c08f190206437b994a91585c71ad78 P 1788a70030a64ae7b5059ac6a1700692: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-2/wal/wals/d1c08f190206437b994a91585c71ad78
I20260926 20:53:54.472719 16292 log.cc:1196] T d1c08f190206437b994a91585c71ad78 P fb366842fb274b3fa8b268cece62eab0: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-1/wal/wals/d1c08f190206437b994a91585c71ad78
I20260926 20:53:54.472962 16291 ts_tablet_manager.cc:1939] T d1c08f190206437b994a91585c71ad78 P 1788a70030a64ae7b5059ac6a1700692: Deleting consensus metadata
I20260926 20:53:54.473049 16292 ts_tablet_manager.cc:1939] T d1c08f190206437b994a91585c71ad78 P fb366842fb274b3fa8b268cece62eab0: Deleting consensus metadata
I20260926 20:53:54.473508 15790 catalog_manager.cc:4841] TS fb366842fb274b3fa8b268cece62eab0 (127.0.98.2:41677): tablet d1c08f190206437b994a91585c71ad78 (table test-table [id=c22ac9799c6e46f596d148c7c23a44c7]) successfully deleted
I20260926 20:53:54.473624 15789 catalog_manager.cc:4841] TS 1788a70030a64ae7b5059ac6a1700692 (127.0.98.3:40147): tablet d1c08f190206437b994a91585c71ad78 (table test-table [id=c22ac9799c6e46f596d148c7c23a44c7]) successfully deleted
W20260926 20:53:54.474707 15791 connection.cc:535] client connection to 127.0.98.1:41173 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 20:53:54.474826 15791 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 20:53:54.475265 16268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:54.475138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41173 (local address 127.0.0.1:41706)
0926 20:53:54.475195 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:54.475230 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 20:53:54.475378 15791 catalog_manager.cc:4584] TS 9a0a759577424f46b7341380af800802 (127.0.98.1:41173): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d1c08f190206437b994a91585c71ad78: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41173: connect: Connection refused (error 111)
I20260926 20:53:54.476263 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 15997
I20260926 20:53:54.482152 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 16133
I20260926 20:53:54.487164 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 15771
I20260926 20:54:14.492898 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43565
--webserver_interface=127.0.98.62
--webserver_port=39329
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:54:14.574132 16294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:54:14.574335 16294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:54:14.574357 16294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:54:14.575512 16294 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 20:54:14.575555 16294 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 20:54:14.575568 16294 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 20:54:14.575579 16294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:54:14.575592 16294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:54:14.575605 16294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:54:14.575618 16294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:54:14.575670 16294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:54:14.577162 16294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=39329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:54:14.577401 16294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:54:14.577656 16294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:54:14.582650 16302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:14.582701 16294 server_base.cc:1034] running on GCE node
W20260926 20:54:14.582656 16299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:14.582651 16300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:14.583106 16294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:54:14.583419 16294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:54:14.584600 16294 hybrid_clock.cc:648] HybridClock initialized: now 1790456054584591 us; error 41 us; skew 500 ppm
Sep 26 20:54:14 dist-test-slave-hwlj krb5kdc[15223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456054, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:54:14.593214 16294 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:54:14.593732 16294 webserver.cc:460] Webserver started at http://127.0.98.62:39329/ using document root <none> and password file <none>
I20260926 20:54:14.594058 16294 fs_manager.cc:359] Metadata directory not provided
I20260926 20:54:14.594144 16294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:54:14.595721 16294 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:54:14.596652 16309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:14.596936 16294 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:54:14.597060 16294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "eddc2af3af4e422187c40b7416070efe"
format_stamp: "Formatted at 2026-09-26 20:53:32 on dist-test-slave-hwlj"
server_key: "06a0ca64fe3568ad0f5001e1dd534c9c"
server_key_iv: "dcbfedebbc2015f628b518203f1b07fc"
server_key_version: "encryptionkey@0"
I20260926 20:54:14.597467 16294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:54:14.608645 16294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:54:14.610667 16294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:54:14.610848 16294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:54:14.615016 16294 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43565
I20260926 20:54:14.615051 16361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43565 every 8 connection(s)
I20260926 20:54:14.615442 16294 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:54:14.617023 16362 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:54:14.617602 16362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe: Bootstrap starting.
I20260926 20:54:14.618002 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 16294
I20260926 20:54:14.618335 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41173
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43565
--webserver_port=36875
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:54:14.626682 16362 log.cc:824] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe: Log is configured to *not* fsync() on all Append() calls
I20260926 20:54:14.628989 16362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:54:14.629307 16362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe: Bootstrap complete.
I20260926 20:54:14.631515 16362 raft_consensus.cc:348] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eddc2af3af4e422187c40b7416070efe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43565 } }
I20260926 20:54:14.631763 16362 raft_consensus.cc:729] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: eddc2af3af4e422187c40b7416070efe, State: Initialized, Role: FOLLOWER
I20260926 20:54:14.631984 16362 consensus_queue.cc:260] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eddc2af3af4e422187c40b7416070efe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43565 } }
I20260926 20:54:14.632079 16362 raft_consensus.cc:388] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:54:14.632157 16362 raft_consensus.cc:482] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:54:14.632217 16362 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe [term 2 FOLLOWER]: Advancing to term 3
I20260926 20:54:14.633215 16362 raft_consensus.cc:504] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eddc2af3af4e422187c40b7416070efe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43565 } }
I20260926 20:54:14.633345 16362 leader_election.cc:302] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eddc2af3af4e422187c40b7416070efe; no voters:
I20260926 20:54:14.633560 16362 leader_election.cc:288] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 20:54:14.633595 16366 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe [term 3 FOLLOWER]: Leader election won for term 3
I20260926 20:54:14.633710 16366 raft_consensus.cc:686] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe [term 3 LEADER]: Becoming Leader. State: Replica: eddc2af3af4e422187c40b7416070efe, State: Running, Role: LEADER
I20260926 20:54:14.633843 16362 sys_catalog.cc:564] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:54:14.633859 16366 consensus_queue.cc:237] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eddc2af3af4e422187c40b7416070efe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43565 } }
I20260926 20:54:14.634320 16367 sys_catalog.cc:455] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "eddc2af3af4e422187c40b7416070efe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eddc2af3af4e422187c40b7416070efe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43565 } } }
I20260926 20:54:14.634410 16367 sys_catalog.cc:458] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe [sys.catalog]: This master's current role is: LEADER
I20260926 20:54:14.634697 16368 sys_catalog.cc:455] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe [sys.catalog]: SysCatalogTable state changed. Reason: New leader eddc2af3af4e422187c40b7416070efe. Latest consensus state: current_term: 3 leader_uuid: "eddc2af3af4e422187c40b7416070efe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eddc2af3af4e422187c40b7416070efe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43565 } } }
I20260926 20:54:14.634789 16368 sys_catalog.cc:458] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe [sys.catalog]: This master's current role is: LEADER
I20260926 20:54:14.635020 16373 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:54:14.635878 16373 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:54:14.636215 16373 catalog_manager.cc:1259] Loaded cluster ID: f1e596f2c2db459a95634083ca1d1205
I20260926 20:54:14.636257 16373 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:54:14.637590 16373 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:54:14.637800 16373 catalog_manager.cc:5879] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe: Loaded TSK: 0
I20260926 20:54:14.637858 16373 catalog_manager.cc:5879] T 00000000000000000000000000000000 P eddc2af3af4e422187c40b7416070efe: Loaded TSK: 1
F20260926 20:54:14.638212 16373 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb9ba1a59ad google::LogMessage::Fail() at ??:0
@ 0x7fb9ba1a888a google::LogMessage::SendToLog() at ??:0
@ 0x7fb9ba1a54de google::LogMessage::Flush() at ??:0
@ 0x7fb9ba1a6a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb9bc912104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fb9bc90fd41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fb9c20ff1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fb9c20ff48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fb9c20ff594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fb9c20c5c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fb9c20cfad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb9bc0730d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb9bc068954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb9bcecc6db start_thread at ??:0
@ 0x7fb9b8ca361f clone at ??:0
W20260926 20:54:14.726990 16364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:54:14.727277 16364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:54:14.727324 16364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:54:14.728787 16364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:54:14.728879 16364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:54:14.728942 16364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:54:14.730667 16364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41173
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36875
--tserver_master_addrs=127.0.98.62:43565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:54:14.730995 16364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:54:14.731259 16364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:54:14.737809 16405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:14.737814 16404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:14.738166 16364 server_base.cc:1034] running on GCE node
W20260926 20:54:14.738320 16407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:14.738548 16364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:54:14.738873 16364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:54:14.740058 16364 hybrid_clock.cc:648] HybridClock initialized: now 1790456054740036 us; error 38 us; skew 500 ppm
Sep 26 20:54:14 dist-test-slave-hwlj krb5kdc[15223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456054, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:54:14.742815 16364 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 20:54:14.743335 16364 webserver.cc:460] Webserver started at http://127.0.98.1:36875/ using document root <none> and password file <none>
I20260926 20:54:14.743605 16364 fs_manager.cc:359] Metadata directory not provided
I20260926 20:54:14.743681 16364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:54:14.745223 16364 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:54:14.746022 16414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:14.746292 16364 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:54:14.746443 16364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-0/wal
uuid: "9a0a759577424f46b7341380af800802"
format_stamp: "Formatted at 2026-09-26 20:53:33 on dist-test-slave-hwlj"
server_key: "ee6e85469a0994cf33e261b7008e0ed3"
server_key_iv: "e5e60cc76a891fff44870d8472b99adc"
server_key_version: "encryptionkey@0"
I20260926 20:54:14.746845 16364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:54:14.776957 16364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:54:14.780051 16364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:54:14.780217 16364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:54:14.780483 16364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:54:14.780983 16421 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 20:54:14.783108 16364 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 20:54:14.783187 16364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 20:54:14.783248 16364 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 20:54:14.784158 16364 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 20:54:14.784266 16364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.001s
I20260926 20:54:14.784718 16421 tablet_bootstrap.cc:492] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802: Bootstrap starting.
I20260926 20:54:14.792263 16364 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41173
I20260926 20:54:14.792853 16364 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:54:14.794337 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 16364
I20260926 20:54:14.794642 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41677
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43565
--webserver_port=46305
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:54:14.796702 16421 log.cc:824] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802: Log is configured to *not* fsync() on all Append() calls
I20260926 20:54:14.797101 16528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41173 every 8 connection(s)
I20260926 20:54:14.799575 16421 tablet_bootstrap.cc:492] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:54:14.799947 16421 tablet_bootstrap.cc:492] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802: Bootstrap complete.
I20260926 20:54:14.801146 16421 ts_tablet_manager.cc:1397] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.012s
I20260926 20:54:14.802623 16421 raft_consensus.cc:348] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb366842fb274b3fa8b268cece62eab0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41677 } } peers { permanent_uuid: "1788a70030a64ae7b5059ac6a1700692" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40147 } } peers { permanent_uuid: "9a0a759577424f46b7341380af800802" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41173 } }
I20260926 20:54:14.802954 16421 raft_consensus.cc:729] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a0a759577424f46b7341380af800802, State: Initialized, Role: FOLLOWER
I20260926 20:54:14.803164 16421 consensus_queue.cc:260] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb366842fb274b3fa8b268cece62eab0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41677 } } peers { permanent_uuid: "1788a70030a64ae7b5059ac6a1700692" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40147 } } peers { permanent_uuid: "9a0a759577424f46b7341380af800802" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41173 } }
I20260926 20:54:14.803488 16421 ts_tablet_manager.cc:1428] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
W20260926 20:54:14.849833 16531 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:54:14.793738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.1:55597)
0926 20:54:14.793978 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:14.794032 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:14.794698 (+ 666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:14.849626 (+ 54928us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43565: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":119,"threads_started":1}
W20260926 20:54:14.850152 16529 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43565 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:43565: Client connection negotiation failed: client connection to 127.0.98.62:43565: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 20:54:14.850569 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:14.850432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.1:38761)
0926 20:54:14.850472 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:14.850520 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260926 20:54:14.850922 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:14.850821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.1:36265)
0926 20:54:14.850860 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:14.850900 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260926 20:54:14.851064 16529 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 20:54:14.879060 16532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:54:14.879262 16532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:54:14.879285 16532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:54:14.880535 16532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:54:14.880584 16532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:54:14.880609 16532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:54:14.882023 16532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41677
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46305
--tserver_master_addrs=127.0.98.62:43565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:54:14.882225 16532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:54:14.882402 16532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:54:14.887539 16539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:14.887552 16542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:14.887606 16532 server_base.cc:1034] running on GCE node
W20260926 20:54:14.887565 16540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:14.887995 16532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:54:14.888247 16532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:54:14.889390 16532 hybrid_clock.cc:648] HybridClock initialized: now 1790456054889381 us; error 31 us; skew 500 ppm
Sep 26 20:54:14 dist-test-slave-hwlj krb5kdc[15223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456054, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:54:14.892011 16532 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 20:54:14.892498 16532 webserver.cc:460] Webserver started at http://127.0.98.2:46305/ using document root <none> and password file <none>
I20260926 20:54:14.892751 16532 fs_manager.cc:359] Metadata directory not provided
I20260926 20:54:14.892827 16532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:54:14.894173 16532 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:54:14.894824 16549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:14.895015 16532 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:54:14.895125 16532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-1/wal
uuid: "fb366842fb274b3fa8b268cece62eab0"
format_stamp: "Formatted at 2026-09-26 20:53:33 on dist-test-slave-hwlj"
server_key: "d3dde8a914d4b2b7e8e057f788e4ab75"
server_key_iv: "f98a5af4f03254e4fe9e351b774e53f4"
server_key_version: "encryptionkey@0"
I20260926 20:54:14.895485 16532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:54:14.918354 16532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:54:14.920799 16532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:54:14.921000 16532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:54:14.921262 16532 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:54:14.921607 16532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:54:14.921667 16532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:14.921711 16532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:54:14.921741 16532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:14.927713 16532 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41677
I20260926 20:54:14.927776 16662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41677 every 8 connection(s)
I20260926 20:54:14.928156 16532 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
I20260926 20:54:14.929296 16665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:14.928873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.2:42875)
0926 20:54:14.929065 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:14.929140 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":106,"threads_started":1}
W20260926 20:54:14.929726 16663 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43565 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:43565: Client connection negotiation failed: client connection to 127.0.98.62:43565: connect: Connection refused (error 111)
I20260926 20:54:14.930007 16665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:14.929920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.2:33533)
0926 20:54:14.929964 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:14.929990 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 20:54:14.930290 16665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:14.930219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.2:52743)
0926 20:54:14.930256 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:14.930277 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260926 20:54:14.930404 16663 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 20:54:14.930496 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 16532
I20260926 20:54:14.930776 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40147
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43565
--webserver_port=41589
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:54:15.013625 16666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:54:15.013873 16666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:54:15.013902 16666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:54:15.015066 16666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:54:15.015108 16666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:54:15.015136 16666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:54:15.016721 16666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40147
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41589
--tserver_master_addrs=127.0.98.62:43565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:54:15.016942 16666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:54:15.017118 16666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:54:15.022527 16674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:15.022537 16671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:15.022537 16672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:15.022614 16666 server_base.cc:1034] running on GCE node
I20260926 20:54:15.023010 16666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:54:15.023232 16666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:54:15.024430 16666 hybrid_clock.cc:648] HybridClock initialized: now 1790456055024417 us; error 52 us; skew 500 ppm
Sep 26 20:54:15 dist-test-slave-hwlj krb5kdc[15223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:54:15.027097 16666 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 20:54:15.027562 16666 webserver.cc:460] Webserver started at http://127.0.98.3:41589/ using document root <none> and password file <none>
I20260926 20:54:15.027901 16666 fs_manager.cc:359] Metadata directory not provided
I20260926 20:54:15.027978 16666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:54:15.029387 16666 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:54:15.030133 16681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:15.030369 16666 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:54:15.030478 16666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-2/wal
uuid: "1788a70030a64ae7b5059ac6a1700692"
format_stamp: "Formatted at 2026-09-26 20:53:33 on dist-test-slave-hwlj"
server_key: "dfbb201cea7e108042799806bed68557"
server_key_iv: "894f56d5750433eda96deba6a9e04248"
server_key_version: "encryptionkey@0"
I20260926 20:54:15.030894 16666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:54:15.059105 16666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:54:15.061228 16666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:54:15.061414 16666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:54:15.061674 16666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:54:15.062001 16666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:54:15.062057 16666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:15.062098 16666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:54:15.062125 16666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:15.068251 16666 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40147
I20260926 20:54:15.068293 16794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40147 every 8 connection(s)
I20260926 20:54:15.068663 16666 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
I20260926 20:54:15.069700 16797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:15.069353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.3:45921)
0926 20:54:15.069496 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:15.069566 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":65,"threads_started":1}
W20260926 20:54:15.070134 16795 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43565 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:43565: Client connection negotiation failed: client connection to 127.0.98.62:43565: connect: Connection refused (error 111)
I20260926 20:54:15.070393 16797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:15.070304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.3:41291)
0926 20:54:15.070348 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:15.070376 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 20:54:15.070629 16797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:15.070578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.3:43723)
0926 20:54:15.070599 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:15.070618 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260926 20:54:15.070737 16795 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 20:54:15.076292 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 16666
I20260926 20:54:15.851598 16799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:15.851405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.1:35935)
0926 20:54:15.851520 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:15.851555 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":58,"threads_started":1}
I20260926 20:54:15.930373 16800 raft_consensus.cc:482] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:54:15.930490 16800 raft_consensus.cc:504] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb366842fb274b3fa8b268cece62eab0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41677 } } peers { permanent_uuid: "1788a70030a64ae7b5059ac6a1700692" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40147 } } peers { permanent_uuid: "9a0a759577424f46b7341380af800802" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41173 } }
I20260926 20:54:15.930845 16800 leader_election.cc:288] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fb366842fb274b3fa8b268cece62eab0 (127.0.98.2:41677), 1788a70030a64ae7b5059ac6a1700692 (127.0.98.3:40147)
I20260926 20:54:15.931108 16801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:15.930849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.2:37461)
0926 20:54:15.930982 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:15.931050 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":55,"threads_started":1}
Sep 26 20:54:15 dist-test-slave-hwlj krb5kdc[15223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456054, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 26 20:54:15 dist-test-slave-hwlj krb5kdc[15223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456054, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260926 20:54:15.942708 16802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:15.931033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40147 (local address 127.0.98.1:33793)
0926 20:54:15.931174 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:15.931188 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:15.931227 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:15.932504 (+ 1277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:15.932505 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:15.932776 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:54:15.933032 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:15.933041 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:15.934449 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:15.934450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:15.935334 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:15.935337 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:15.936082 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:15.936083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:15.936201 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:15.936861 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:54:15.936869 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:54:15.938853 (+ 1984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:54:15.941052 (+ 2199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:54:15.941053 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:54:15.941053 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:54:15.941225 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:54:15.941479 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:54:15.941480 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:54:15.941481 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:54:15.941551 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:54:15.941970 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:54:15.941979 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:54:15.942187 (+ 208us) client_negotiation.cc:769] Sending connection context
0926 20:54:15.942334 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 20:54:15.942581 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":86,"thread_start_us":47,"threads_started":1}
I20260926 20:54:15.942754 16803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:15.931242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33793 (local address 127.0.98.3:40147)
0926 20:54:15.931486 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:15.931502 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:15.931531 (+ 29us) server_negotiation.cc:407] Connection header received
0926 20:54:15.932125 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:15.932128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:15.932190 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:15.932353 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:15.933154 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:15.934299 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:15.935439 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:15.935986 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:15.936101 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:15.938991 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:15.938995 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:15.938995 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:54:15.939005 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:15.940973 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:15.941320 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:15.941322 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:15.941323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:15.941402 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:15.941611 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:15.941611 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:15.941612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:15.941866 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:15.941946 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:15.942467 (+ 521us) server_negotiation.cc:299] Negotiation successful
0926 20:54:15.942621 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":68,"threads_started":1}
I20260926 20:54:15.942746 16804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:15.931295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55577 (local address 127.0.98.2:41677)
0926 20:54:15.931486 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:15.931498 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:15.931537 (+ 39us) server_negotiation.cc:407] Connection header received
0926 20:54:15.932125 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:15.932128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:15.932190 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:15.932353 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:15.933178 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:15.934299 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:15.935513 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:15.936002 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:15.936129 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:15.938541 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:15.938547 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:15.938547 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:54:15.938561 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:15.940686 (+ 2125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:15.941127 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:15.941130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:15.941130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:15.941216 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:15.941524 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:15.941525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:15.941525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:15.941866 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:15.941946 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:15.942482 (+ 536us) server_negotiation.cc:299] Negotiation successful
0926 20:54:15.942620 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":46,"threads_started":1}
I20260926 20:54:15.942767 16799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:15.931033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41677 (local address 127.0.98.1:55577)
0926 20:54:15.931090 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:15.931102 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:15.931156 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:15.932411 (+ 1255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:15.932414 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:15.932790 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:54:15.933048 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:15.933052 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:15.934424 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:15.934425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:15.935404 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:15.935408 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:15.936108 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:15.936109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:15.936201 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:15.936879 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:54:15.936884 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:54:15.938423 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:54:15.940769 (+ 2346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:54:15.940776 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:54:15.940778 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:54:15.941033 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:54:15.941294 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:54:15.941295 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:54:15.941295 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:54:15.941451 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:54:15.941970 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:54:15.941979 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:54:15.942187 (+ 208us) client_negotiation.cc:769] Sending connection context
0926 20:54:15.942334 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 20:54:15.942638 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":248}
W20260926 20:54:15.944144 16416 leader_election.cc:341] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1788a70030a64ae7b5059ac6a1700692 (127.0.98.3:40147): Not found: Tablet not found: d1c08f190206437b994a91585c71ad78
W20260926 20:54:15.944276 16417 leader_election.cc:341] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fb366842fb274b3fa8b268cece62eab0 (127.0.98.2:41677): Not found: Tablet not found: d1c08f190206437b994a91585c71ad78
I20260926 20:54:15.944360 16417 leader_election.cc:302] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a0a759577424f46b7341380af800802; no voters: 1788a70030a64ae7b5059ac6a1700692, fb366842fb274b3fa8b268cece62eab0
I20260926 20:54:15.944484 16800 raft_consensus.cc:2726] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 20:54:16.071333 16805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:16.071068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.3:53049)
0926 20:54:16.071236 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:16.071277 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":68,"threads_started":1}
I20260926 20:54:16.852397 16806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:16.852134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.1:56151)
0926 20:54:16.852318 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:16.852353 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":101,"threads_started":1}
I20260926 20:54:16.932041 16807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:16.931831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.2:37811)
0926 20:54:16.931963 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:16.932000 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":78,"threads_started":1}
I20260926 20:54:17.072049 16808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:17.071830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.3:37175)
0926 20:54:17.071952 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:17.071996 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":59,"threads_started":1}
I20260926 20:54:17.853109 16809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:17.852871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.1:58879)
0926 20:54:17.853019 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:17.853062 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":62,"threads_started":1}
I20260926 20:54:17.931106 16810 raft_consensus.cc:482] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:54:17.931207 16810 raft_consensus.cc:504] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb366842fb274b3fa8b268cece62eab0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41677 } } peers { permanent_uuid: "1788a70030a64ae7b5059ac6a1700692" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40147 } } peers { permanent_uuid: "9a0a759577424f46b7341380af800802" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41173 } }
I20260926 20:54:17.931443 16810 leader_election.cc:288] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fb366842fb274b3fa8b268cece62eab0 (127.0.98.2:41677), 1788a70030a64ae7b5059ac6a1700692 (127.0.98.3:40147)
W20260926 20:54:17.931969 16416 leader_election.cc:341] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1788a70030a64ae7b5059ac6a1700692 (127.0.98.3:40147): Not found: Tablet not found: d1c08f190206437b994a91585c71ad78
W20260926 20:54:17.932092 16417 leader_election.cc:341] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fb366842fb274b3fa8b268cece62eab0 (127.0.98.2:41677): Not found: Tablet not found: d1c08f190206437b994a91585c71ad78
I20260926 20:54:17.932159 16417 leader_election.cc:302] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a0a759577424f46b7341380af800802; no voters: 1788a70030a64ae7b5059ac6a1700692, fb366842fb274b3fa8b268cece62eab0
I20260926 20:54:17.932229 16810 raft_consensus.cc:2726] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 20:54:17.932621 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:17.932451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.2:39187)
0926 20:54:17.932552 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:17.932584 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":52,"threads_started":1}
I20260926 20:54:18.072773 16812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:18.072535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.3:45491)
0926 20:54:18.072672 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:18.072711 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":71,"threads_started":1}
I20260926 20:54:18.853865 16813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:18.853584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.1:58557)
0926 20:54:18.853776 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:18.853817 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":64,"threads_started":1}
I20260926 20:54:18.933478 16814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:18.933155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.2:54225)
0926 20:54:18.933303 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:18.933399 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":64,"threads_started":1}
I20260926 20:54:19.073494 16815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:19.073282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.3:47317)
0926 20:54:19.073411 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:19.073451 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":63,"threads_started":1}
I20260926 20:54:19.854671 16816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:19.854432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.1:41311)
0926 20:54:19.854575 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:19.854619 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":65,"threads_started":1}
I20260926 20:54:19.934340 16817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:19.934049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.2:40261)
0926 20:54:19.934241 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:19.934293 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":64,"threads_started":1}
I20260926 20:54:20.074280 16818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:20.074035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.3:39371)
0926 20:54:20.074191 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:20.074234 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":58,"threads_started":1}
I20260926 20:54:20.855389 16819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:20.855169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.1:38521)
0926 20:54:20.855299 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:20.855338 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":71,"threads_started":1}
I20260926 20:54:20.935025 16820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:20.934792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.2:56145)
0926 20:54:20.934929 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:20.934970 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":74,"threads_started":1}
I20260926 20:54:21.075121 16821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:21.074858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.3:46145)
0926 20:54:21.075026 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:21.075071 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":89,"threads_started":1}
I20260926 20:54:21.118482 16822 raft_consensus.cc:482] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:54:21.118623 16822 raft_consensus.cc:504] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb366842fb274b3fa8b268cece62eab0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41677 } } peers { permanent_uuid: "1788a70030a64ae7b5059ac6a1700692" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40147 } } peers { permanent_uuid: "9a0a759577424f46b7341380af800802" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41173 } }
I20260926 20:54:21.118842 16822 leader_election.cc:288] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fb366842fb274b3fa8b268cece62eab0 (127.0.98.2:41677), 1788a70030a64ae7b5059ac6a1700692 (127.0.98.3:40147)
W20260926 20:54:21.119256 16417 leader_election.cc:341] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fb366842fb274b3fa8b268cece62eab0 (127.0.98.2:41677): Not found: Tablet not found: d1c08f190206437b994a91585c71ad78
W20260926 20:54:21.119335 16416 leader_election.cc:341] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1788a70030a64ae7b5059ac6a1700692 (127.0.98.3:40147): Not found: Tablet not found: d1c08f190206437b994a91585c71ad78
I20260926 20:54:21.119410 16416 leader_election.cc:302] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a0a759577424f46b7341380af800802; no voters: 1788a70030a64ae7b5059ac6a1700692, fb366842fb274b3fa8b268cece62eab0
I20260926 20:54:21.119491 16822 raft_consensus.cc:2726] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 20:54:21.856171 16823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:21.855965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.1:52969)
0926 20:54:21.856087 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:21.856123 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":63,"threads_started":1}
I20260926 20:54:21.935835 16824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:21.935543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.2:48089)
0926 20:54:21.935682 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:21.935722 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":61,"threads_started":1}
I20260926 20:54:22.075914 16825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:22.075640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.3:48795)
0926 20:54:22.075829 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:22.075867 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":60,"threads_started":1}
I20260926 20:54:22.856931 16826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:22.856706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.1:38943)
0926 20:54:22.856842 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:22.856883 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":84,"threads_started":1}
I20260926 20:54:22.936561 16827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:22.936340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.2:57203)
0926 20:54:22.936468 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:22.936513 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":61,"threads_started":1}
I20260926 20:54:23.076627 16828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:23.076401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.3:36881)
0926 20:54:23.076540 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:23.076579 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":74,"threads_started":1}
I20260926 20:54:23.857679 16829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:23.857443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.1:48811)
0926 20:54:23.857589 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:23.857631 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":76,"threads_started":1}
I20260926 20:54:23.937379 16830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:23.937124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.2:60383)
0926 20:54:23.937279 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:23.937330 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":64,"threads_started":1}
I20260926 20:54:24.077325 16831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:24.077110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.3:52953)
0926 20:54:24.077240 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:24.077276 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":61,"threads_started":1}
I20260926 20:54:24.858500 16832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:24.858256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.1:49345)
0926 20:54:24.858396 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:24.858446 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":61,"threads_started":1}
I20260926 20:54:24.938123 16833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:24.937898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.2:40855)
0926 20:54:24.938028 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:24.938076 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":78,"threads_started":1}
I20260926 20:54:25.011068 16834 raft_consensus.cc:482] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:54:25.011163 16834 raft_consensus.cc:504] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb366842fb274b3fa8b268cece62eab0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41677 } } peers { permanent_uuid: "1788a70030a64ae7b5059ac6a1700692" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40147 } } peers { permanent_uuid: "9a0a759577424f46b7341380af800802" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41173 } }
I20260926 20:54:25.011372 16834 leader_election.cc:288] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fb366842fb274b3fa8b268cece62eab0 (127.0.98.2:41677), 1788a70030a64ae7b5059ac6a1700692 (127.0.98.3:40147)
W20260926 20:54:25.011802 16417 leader_election.cc:341] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fb366842fb274b3fa8b268cece62eab0 (127.0.98.2:41677): Not found: Tablet not found: d1c08f190206437b994a91585c71ad78
W20260926 20:54:25.011957 16416 leader_election.cc:341] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1788a70030a64ae7b5059ac6a1700692 (127.0.98.3:40147): Not found: Tablet not found: d1c08f190206437b994a91585c71ad78
I20260926 20:54:25.012022 16416 leader_election.cc:302] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a0a759577424f46b7341380af800802; no voters: 1788a70030a64ae7b5059ac6a1700692, fb366842fb274b3fa8b268cece62eab0
I20260926 20:54:25.012110 16834 raft_consensus.cc:2726] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 20:54:25.078075 16835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:25.077827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.3:35079)
0926 20:54:25.077984 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:25.078026 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":90,"threads_started":1}
I20260926 20:54:25.859297 16836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:25.859020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.1:36975)
0926 20:54:25.859187 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:25.859227 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":66,"threads_started":1}
I20260926 20:54:25.938849 16837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:25.938594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.2:50745)
0926 20:54:25.938754 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:25.938804 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":63,"threads_started":1}
I20260926 20:54:26.078745 16838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:26.078535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.3:42303)
0926 20:54:26.078661 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:26.078697 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":60,"threads_started":1}
I20260926 20:54:26.580631 16839 raft_consensus.cc:482] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:54:26.580724 16839 raft_consensus.cc:504] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb366842fb274b3fa8b268cece62eab0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41677 } } peers { permanent_uuid: "1788a70030a64ae7b5059ac6a1700692" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40147 } } peers { permanent_uuid: "9a0a759577424f46b7341380af800802" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41173 } }
I20260926 20:54:26.580895 16839 leader_election.cc:288] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fb366842fb274b3fa8b268cece62eab0 (127.0.98.2:41677), 1788a70030a64ae7b5059ac6a1700692 (127.0.98.3:40147)
W20260926 20:54:26.581259 16417 leader_election.cc:341] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fb366842fb274b3fa8b268cece62eab0 (127.0.98.2:41677): Not found: Tablet not found: d1c08f190206437b994a91585c71ad78
W20260926 20:54:26.581373 16416 leader_election.cc:341] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1788a70030a64ae7b5059ac6a1700692 (127.0.98.3:40147): Not found: Tablet not found: d1c08f190206437b994a91585c71ad78
I20260926 20:54:26.581418 16416 leader_election.cc:302] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a0a759577424f46b7341380af800802; no voters: 1788a70030a64ae7b5059ac6a1700692, fb366842fb274b3fa8b268cece62eab0
I20260926 20:54:26.581511 16839 raft_consensus.cc:2726] T d1c08f190206437b994a91585c71ad78 P 9a0a759577424f46b7341380af800802 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 20:54:26.860152 16840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:26.859936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.1:36785)
0926 20:54:26.860067 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:26.860103 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":67,"threads_started":1}
I20260926 20:54:26.939601 16841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:26.939345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.2:56687)
0926 20:54:26.939512 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:26.939554 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":92,"threads_started":1}
I20260926 20:54:27.079470 16842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:27.079233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.3:51519)
0926 20:54:27.079367 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:27.079412 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":88,"threads_started":1}
I20260926 20:54:27.861016 16843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:27.860686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.1:47971)
0926 20:54:27.860918 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:27.860964 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":146,"threads_started":1}
I20260926 20:54:27.940416 16844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:27.940185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.2:43133)
0926 20:54:27.940312 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:27.940355 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":60,"threads_started":1}
I20260926 20:54:28.080360 16845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:28.080124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.3:34019)
0926 20:54:28.080268 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:28.080311 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":80,"threads_started":1}
I20260926 20:54:28.861800 16846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:28.861573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.1:50831)
0926 20:54:28.861709 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:28.861751 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":64,"threads_started":1}
I20260926 20:54:28.941180 16847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:28.940965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.2:37909)
0926 20:54:28.941097 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:28.941134 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":62,"threads_started":1}
I20260926 20:54:29.081187 16848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:29.080934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.3:39407)
0926 20:54:29.081097 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:29.081137 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":79,"threads_started":1}
I20260926 20:54:29.862649 16849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:29.862421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.1:37263)
0926 20:54:29.862560 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:29.862601 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":59,"threads_started":1}
I20260926 20:54:29.941984 16850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:29.941726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.2:60505)
0926 20:54:29.941882 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:29.941939 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":76,"threads_started":1}
I20260926 20:54:30.082052 16851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:30.081854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43565 (local address 127.0.98.3:41555)
0926 20:54:30.081973 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:30.082009 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":60,"threads_started":1}
I20260926 20:54:30.086210 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 16364
I20260926 20:54:30.090797 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 16532
I20260926 20:54:30.094822 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 16666
2026-09-26T20:54:30Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58455 ms)
[----------] 30 tests from SecurityITest (344577 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:54:30 dist-test-slave-hwlj krb5kdc[16854](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:54:30 dist-test-slave-hwlj krb5kdc[16854](info): set up 2 sockets
Sep 26 20:54:30 dist-test-slave-hwlj krb5kdc[16854](info): commencing operation
krb5kdc: starting...
W20260926 20:54:31.155486 392 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/krb5kdc/test-user.keytab.
Sep 26 20:54:31 dist-test-slave-hwlj krb5kdc[16854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:54:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:54:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 20:54:31.282114 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40493
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40493
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:54:31.362138 16867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:54:31.362344 16867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:54:31.362366 16867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:54:31.363539 16867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:54:31.363580 16867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:54:31.363595 16867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:54:31.363637 16867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:54:31.363667 16867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:54:31.365156 16867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:54:31.365410 16867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:54:31.365638 16867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:54:31.370651 16872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:31.370674 16875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:31.370704 16873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:31.370702 16867 server_base.cc:1034] running on GCE node
I20260926 20:54:31.371158 16867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:54:31.371447 16867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:54:31.372611 16867 hybrid_clock.cc:648] HybridClock initialized: now 1790456071372598 us; error 31 us; skew 500 ppm
Sep 26 20:54:31 dist-test-slave-hwlj krb5kdc[16854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:54:31.375135 16867 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:54:31.375610 16867 webserver.cc:460] Webserver started at http://127.0.98.62:33195/ using document root <none> and password file <none>
I20260926 20:54:31.375892 16867 fs_manager.cc:359] Metadata directory not provided
I20260926 20:54:31.375960 16867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:54:31.376118 16867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:54:31.377159 16867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/master-0/data/instance:
uuid: "bb9301410c9c45788c3011fd4f40df08"
format_stamp: "Formatted at 2026-09-26 20:54:31 on dist-test-slave-hwlj"
server_key: "3f3849167ab218152455610f4198b66a"
server_key_iv: "86b6a477fe7d36a886dc878dacb8e444"
server_key_version: "encryptionkey@0"
I20260926 20:54:31.377526 16867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/master-0/wal/instance:
uuid: "bb9301410c9c45788c3011fd4f40df08"
format_stamp: "Formatted at 2026-09-26 20:54:31 on dist-test-slave-hwlj"
server_key: "3f3849167ab218152455610f4198b66a"
server_key_iv: "86b6a477fe7d36a886dc878dacb8e444"
server_key_version: "encryptionkey@0"
I20260926 20:54:31.378875 16867 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:54:31.379608 16882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:31.379808 16867 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:54:31.379938 16867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "bb9301410c9c45788c3011fd4f40df08"
format_stamp: "Formatted at 2026-09-26 20:54:31 on dist-test-slave-hwlj"
server_key: "3f3849167ab218152455610f4198b66a"
server_key_iv: "86b6a477fe7d36a886dc878dacb8e444"
server_key_version: "encryptionkey@0"
I20260926 20:54:31.380038 16867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:54:31.401860 16867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:54:31.404299 16867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:54:31.404472 16867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:54:31.408532 16867 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40493
I20260926 20:54:31.408571 16934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40493 every 8 connection(s)
I20260926 20:54:31.408957 16867 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:54:31.409788 16935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:54:31.412067 16935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb9301410c9c45788c3011fd4f40df08: Bootstrap starting.
I20260926 20:54:31.412645 16935 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb9301410c9c45788c3011fd4f40df08: Neither blocks nor log segments found. Creating new log.
I20260926 20:54:31.412938 16935 log.cc:824] T 00000000000000000000000000000000 P bb9301410c9c45788c3011fd4f40df08: Log is configured to *not* fsync() on all Append() calls
I20260926 20:54:31.413663 16935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb9301410c9c45788c3011fd4f40df08: No bootstrap required, opened a new log
I20260926 20:54:31.415304 16935 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb9301410c9c45788c3011fd4f40df08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb9301410c9c45788c3011fd4f40df08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40493 } }
I20260926 20:54:31.415445 16935 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb9301410c9c45788c3011fd4f40df08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:54:31.415467 16935 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb9301410c9c45788c3011fd4f40df08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb9301410c9c45788c3011fd4f40df08, State: Initialized, Role: FOLLOWER
I20260926 20:54:31.415642 16935 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb9301410c9c45788c3011fd4f40df08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb9301410c9c45788c3011fd4f40df08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40493 } }
I20260926 20:54:31.415706 16935 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb9301410c9c45788c3011fd4f40df08 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:54:31.415750 16935 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb9301410c9c45788c3011fd4f40df08 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:54:31.415833 16935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb9301410c9c45788c3011fd4f40df08 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:54:31.416399 16935 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb9301410c9c45788c3011fd4f40df08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb9301410c9c45788c3011fd4f40df08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40493 } }
I20260926 20:54:31.416504 16935 leader_election.cc:302] T 00000000000000000000000000000000 P bb9301410c9c45788c3011fd4f40df08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb9301410c9c45788c3011fd4f40df08; no voters:
I20260926 20:54:31.416684 16935 leader_election.cc:288] T 00000000000000000000000000000000 P bb9301410c9c45788c3011fd4f40df08 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:54:31.416743 16938 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb9301410c9c45788c3011fd4f40df08 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:54:31.416914 16938 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb9301410c9c45788c3011fd4f40df08 [term 1 LEADER]: Becoming Leader. State: Replica: bb9301410c9c45788c3011fd4f40df08, State: Running, Role: LEADER
I20260926 20:54:31.416942 16935 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb9301410c9c45788c3011fd4f40df08 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:54:31.417017 16938 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb9301410c9c45788c3011fd4f40df08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb9301410c9c45788c3011fd4f40df08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40493 } }
I20260926 20:54:31.417040 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 16867
I20260926 20:54:31.417133 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/master-0/wal/instance
I20260926 20:54:31.417331 392 external_mini_cluster.cc:1428] Setting key 1512633c5098323f0e7f4b256bb29c40
I20260926 20:54:31.417548 16939 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb9301410c9c45788c3011fd4f40df08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb9301410c9c45788c3011fd4f40df08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb9301410c9c45788c3011fd4f40df08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40493 } } }
I20260926 20:54:31.417634 16939 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb9301410c9c45788c3011fd4f40df08 [sys.catalog]: This master's current role is: LEADER
I20260926 20:54:31.417814 16940 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb9301410c9c45788c3011fd4f40df08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb9301410c9c45788c3011fd4f40df08. Latest consensus state: current_term: 1 leader_uuid: "bb9301410c9c45788c3011fd4f40df08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb9301410c9c45788c3011fd4f40df08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40493 } } }
I20260926 20:54:31.417860 16947 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:54:31.417882 16940 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb9301410c9c45788c3011fd4f40df08 [sys.catalog]: This master's current role is: LEADER
I20260926 20:54:31.418418 16947 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:54:31.421052 16947 catalog_manager.cc:1347] Generated new cluster ID: c7cee5c1d1694159b8295c7eb6e39fac
I20260926 20:54:31.421108 16947 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 20:54:31 dist-test-slave-hwlj krb5kdc[16854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:54:31.428802 16950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:31.417957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49184 (local address 127.0.98.62:40493)
0926 20:54:31.418139 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:31.418148 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:31.418413 (+ 265us) server_negotiation.cc:407] Connection header received
0926 20:54:31.418539 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:31.418546 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:31.418616 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:31.418757 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:31.419602 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:31.420593 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:31.421652 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:31.422312 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:31.422438 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:31.425221 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:31.425225 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:31.425225 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:54:31.425238 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:31.427177 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:31.427575 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:31.427578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:31.427578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:31.427698 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:31.427928 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:31.427929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:31.427929 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:31.428210 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:31.428352 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:31.428469 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 20:54:31.428595 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":69,"threads_started":1}
I20260926 20:54:31.440913 16947 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:54:31.441455 16947 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:54:31.445168 16947 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb9301410c9c45788c3011fd4f40df08: Generated new TSK 0
I20260926 20:54:31.445364 16947 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 20:54:31.470600 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40493
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:54:31.551817 16961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:54:31.552032 16961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:54:31.552055 16961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:54:31.553256 16961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:54:31.553303 16961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:54:31.553328 16961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:54:31.554865 16961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:54:31.555124 16961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:54:31.555356 16961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:54:31.560286 16967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:31.560281 16966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:31.560551 16961 server_base.cc:1034] running on GCE node
W20260926 20:54:31.560276 16969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:31.560833 16961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:54:31.561126 16961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:54:31.562306 16961 hybrid_clock.cc:648] HybridClock initialized: now 1790456071562288 us; error 26 us; skew 500 ppm
Sep 26 20:54:31 dist-test-slave-hwlj krb5kdc[16854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:54:31.565001 16961 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 20:54:31.565449 16961 webserver.cc:460] Webserver started at http://127.0.98.1:39637/ using document root <none> and password file <none>
I20260926 20:54:31.565641 16961 fs_manager.cc:359] Metadata directory not provided
I20260926 20:54:31.565681 16961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:54:31.565768 16961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:54:31.566771 16961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/ts-0/data/instance:
uuid: "a594f0a4fde74a269009343cbec332d9"
format_stamp: "Formatted at 2026-09-26 20:54:31 on dist-test-slave-hwlj"
server_key: "231016588356cfcf7153b6a5a20854b2"
server_key_iv: "a4d5795e6c3518332d392ee7dd6fa7e3"
server_key_version: "encryptionkey@0"
I20260926 20:54:31.567201 16961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a594f0a4fde74a269009343cbec332d9"
format_stamp: "Formatted at 2026-09-26 20:54:31 on dist-test-slave-hwlj"
server_key: "231016588356cfcf7153b6a5a20854b2"
server_key_iv: "a4d5795e6c3518332d392ee7dd6fa7e3"
server_key_version: "encryptionkey@0"
I20260926 20:54:31.568606 16961 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 20:54:31.569391 16976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:31.569581 16961 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:54:31.569660 16961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/ts-0/wal
uuid: "a594f0a4fde74a269009343cbec332d9"
format_stamp: "Formatted at 2026-09-26 20:54:31 on dist-test-slave-hwlj"
server_key: "231016588356cfcf7153b6a5a20854b2"
server_key_iv: "a4d5795e6c3518332d392ee7dd6fa7e3"
server_key_version: "encryptionkey@0"
I20260926 20:54:31.569712 16961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:54:31.580955 16961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:54:31.583151 16961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:54:31.583267 16961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:54:31.583467 16961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:54:31.583854 16961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:54:31.583897 16961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:31.583920 16961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:54:31.583932 16961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:31.589740 16961 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33199
I20260926 20:54:31.589854 17089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33199 every 8 connection(s)
I20260926 20:54:31.590152 16961 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:54:31.595853 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 16961
I20260926 20:54:31.595943 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/ts-0/wal/instance
I20260926 20:54:31.596127 392 external_mini_cluster.cc:1428] Setting key 093a3c72a97ce5e55b799c8f88227e98
Sep 26 20:54:31 dist-test-slave-hwlj krb5kdc[16854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:54:31.600131 16950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:31.590982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45973 (local address 127.0.98.62:40493)
0926 20:54:31.591053 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:31.591055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:31.591253 (+ 198us) server_negotiation.cc:407] Connection header received
0926 20:54:31.592275 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:31.592276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:31.592289 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:31.592361 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:31.592972 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:31.593476 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:31.594471 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:31.594691 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:31.594763 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:31.597094 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:31.597096 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:31.597096 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:54:31.597104 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:31.598655 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:31.599166 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:31.599167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:31.599167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:31.599215 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:31.599499 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:31.599500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:31.599500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:31.599599 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:31.599648 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:31.599998 (+ 350us) server_negotiation.cc:299] Negotiation successful
0926 20:54:31.600067 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 20:54:31.600541 17092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:31.590951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40493 (local address 127.0.98.1:45973)
0926 20:54:31.591125 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:31.591167 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:31.592184 (+ 1017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:31.592422 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:31.592426 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:31.592689 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:54:31.592886 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:31.592889 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:31.593556 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:31.593557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:31.594373 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:31.594375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:31.594778 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:31.594779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:31.594879 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:31.595368 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:54:31.595376 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:54:31.597010 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:54:31.598750 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:54:31.598758 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:54:31.598760 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:54:31.599087 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:54:31.599285 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:54:31.599286 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:54:31.599287 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:54:31.599436 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:54:31.599664 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:54:31.599671 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:54:31.599862 (+ 191us) client_negotiation.cc:769] Sending connection context
0926 20:54:31.599977 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 20:54:31.600161 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":98,"threads_started":1}
I20260926 20:54:31.601082 17090 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40493
I20260926 20:54:31.601195 17090 heartbeater.cc:464] Registering TS with master...
I20260926 20:54:31.601373 17090 heartbeater.cc:511] Master 127.0.98.62:40493 requested a full tablet report, sending...
I20260926 20:54:31.601912 16899 ts_manager.cc:194] Registered new tserver with Master: a594f0a4fde74a269009343cbec332d9 (127.0.98.1:33199)
I20260926 20:54:31.602535 16899 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45973
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 20:54:31.624055 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40493
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:54:31.708850 17095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:54:31.709107 17095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:54:31.709149 17095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:54:31.710371 17095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:54:31.710448 17095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:54:31.710492 17095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:54:31.712109 17095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:54:31.712378 17095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:54:31.712610 17095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:54:31.717695 17103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:31.717757 17095 server_base.cc:1034] running on GCE node
W20260926 20:54:31.717695 17100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:31.717715 17101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:31.718228 17095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:54:31.718513 17095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:54:31.719704 17095 hybrid_clock.cc:648] HybridClock initialized: now 1790456071719687 us; error 30 us; skew 500 ppm
Sep 26 20:54:31 dist-test-slave-hwlj krb5kdc[16854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:54:31.722596 17095 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 20:54:31.723104 17095 webserver.cc:460] Webserver started at http://127.0.98.2:42525/ using document root <none> and password file <none>
I20260926 20:54:31.723346 17095 fs_manager.cc:359] Metadata directory not provided
I20260926 20:54:31.723415 17095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:54:31.723551 17095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:54:31.724874 17095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/ts-1/data/instance:
uuid: "321a657e9e764eb5b9a192db8655c2eb"
format_stamp: "Formatted at 2026-09-26 20:54:31 on dist-test-slave-hwlj"
server_key: "c5f25af452fd8bdea3363dfc57ebae06"
server_key_iv: "61ffaf2478bb6af949b18fcd9f47b20c"
server_key_version: "encryptionkey@0"
I20260926 20:54:31.725267 17095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/ts-1/wal/instance:
uuid: "321a657e9e764eb5b9a192db8655c2eb"
format_stamp: "Formatted at 2026-09-26 20:54:31 on dist-test-slave-hwlj"
server_key: "c5f25af452fd8bdea3363dfc57ebae06"
server_key_iv: "61ffaf2478bb6af949b18fcd9f47b20c"
server_key_version: "encryptionkey@0"
I20260926 20:54:31.726689 17095 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:54:31.727671 17110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:31.727916 17095 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:54:31.728031 17095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/ts-1/wal
uuid: "321a657e9e764eb5b9a192db8655c2eb"
format_stamp: "Formatted at 2026-09-26 20:54:31 on dist-test-slave-hwlj"
server_key: "c5f25af452fd8bdea3363dfc57ebae06"
server_key_iv: "61ffaf2478bb6af949b18fcd9f47b20c"
server_key_version: "encryptionkey@0"
I20260926 20:54:31.728130 17095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:54:31.738492 17095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:54:31.741143 17095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:54:31.741369 17095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:54:31.741658 17095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:54:31.742030 17095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:54:31.742092 17095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:31.742138 17095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:54:31.742172 17095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:31.748889 17095 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42279
I20260926 20:54:31.748931 17223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42279 every 8 connection(s)
I20260926 20:54:31.749285 17095 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 20:54:31 dist-test-slave-hwlj krb5kdc[16854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:54:31.758706 16950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:31.750021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60441 (local address 127.0.98.62:40493)
0926 20:54:31.750125 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:31.750128 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:31.750235 (+ 107us) server_negotiation.cc:407] Connection header received
0926 20:54:31.751200 (+ 965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:31.751201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:31.751214 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:31.751280 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:31.751969 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:31.752506 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:31.753534 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:31.753785 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:31.753853 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:31.756049 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:31.756051 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:31.756051 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:54:31.756058 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:31.757400 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:31.757765 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:31.757766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:31.757766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:31.757819 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:31.758050 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:31.758051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:31.758051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:31.758171 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:31.758227 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:31.758533 (+ 306us) server_negotiation.cc:299] Negotiation successful
0926 20:54:31.758601 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 20:54:31.758873 17226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:31.749958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40493 (local address 127.0.98.2:60441)
0926 20:54:31.750125 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:31.750163 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:31.751077 (+ 914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:31.751350 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:31.751354 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:31.751616 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:54:31.751866 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:31.751871 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:31.752592 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:31.752593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:31.753438 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:31.753440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:31.753871 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:31.753872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:31.753975 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:31.754460 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:54:31.754468 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:54:31.755965 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:54:31.757472 (+ 1507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:54:31.757478 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:54:31.757480 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:54:31.757693 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:54:31.757880 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:54:31.757881 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:54:31.757881 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:54:31.757977 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:54:31.758257 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:54:31.758264 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:54:31.758400 (+ 136us) client_negotiation.cc:769] Sending connection context
0926 20:54:31.758498 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 20:54:31.758624 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":64,"threads_started":1}
I20260926 20:54:31.759200 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 17095
I20260926 20:54:31.759327 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/ts-1/wal/instance
I20260926 20:54:31.759512 17224 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40493
I20260926 20:54:31.759549 392 external_mini_cluster.cc:1428] Setting key efd870de78d7a1f4891c17d67dc1842c
I20260926 20:54:31.759611 17224 heartbeater.cc:464] Registering TS with master...
I20260926 20:54:31.759892 17224 heartbeater.cc:511] Master 127.0.98.62:40493 requested a full tablet report, sending...
I20260926 20:54:31.760321 16898 ts_manager.cc:194] Registered new tserver with Master: 321a657e9e764eb5b9a192db8655c2eb (127.0.98.2:42279)
I20260926 20:54:31.760710 16898 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60441
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 20:54:31.786871 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40493
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:54:31.871891 17229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:54:31.872102 17229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:54:31.872126 17229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:54:31.873441 17229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:54:31.873486 17229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:54:31.873514 17229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:54:31.875056 17229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:54:31.875311 17229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:54:31.875553 17229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:54:31.880636 17237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:31.880681 17235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:31.880683 17234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:31.880965 17229 server_base.cc:1034] running on GCE node
I20260926 20:54:31.881234 17229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:54:31.881494 17229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:54:31.882652 17229 hybrid_clock.cc:648] HybridClock initialized: now 1790456071882630 us; error 43 us; skew 500 ppm
Sep 26 20:54:31 dist-test-slave-hwlj krb5kdc[16854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:54:31.885327 17229 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 20:54:31.885715 17229 webserver.cc:460] Webserver started at http://127.0.98.3:45179/ using document root <none> and password file <none>
I20260926 20:54:31.885905 17229 fs_manager.cc:359] Metadata directory not provided
I20260926 20:54:31.885944 17229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:54:31.886030 17229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:54:31.887040 17229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/ts-2/data/instance:
uuid: "a09035df5a744c43b508a66bc06c742f"
format_stamp: "Formatted at 2026-09-26 20:54:31 on dist-test-slave-hwlj"
server_key: "97e4408de4df165c3bd8fe345a75c963"
server_key_iv: "84b7e5fcec4cfaf640f0ddcb6b22f9ac"
server_key_version: "encryptionkey@0"
I20260926 20:54:31.887343 17229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a09035df5a744c43b508a66bc06c742f"
format_stamp: "Formatted at 2026-09-26 20:54:31 on dist-test-slave-hwlj"
server_key: "97e4408de4df165c3bd8fe345a75c963"
server_key_iv: "84b7e5fcec4cfaf640f0ddcb6b22f9ac"
server_key_version: "encryptionkey@0"
I20260926 20:54:31.888652 17229 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:54:31.889379 17244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:31.889631 17229 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 20:54:31.889760 17229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/ts-2/wal
uuid: "a09035df5a744c43b508a66bc06c742f"
format_stamp: "Formatted at 2026-09-26 20:54:31 on dist-test-slave-hwlj"
server_key: "97e4408de4df165c3bd8fe345a75c963"
server_key_iv: "84b7e5fcec4cfaf640f0ddcb6b22f9ac"
server_key_version: "encryptionkey@0"
I20260926 20:54:31.889820 17229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:54:31.903992 17229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:54:31.906255 17229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:54:31.906370 17229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:54:31.906554 17229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:54:31.906860 17229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:54:31.906893 17229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:31.906945 17229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:54:31.906972 17229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:31.912827 17229 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39807
I20260926 20:54:31.912930 17357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39807 every 8 connection(s)
I20260926 20:54:31.913218 17229 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 20:54:31 dist-test-slave-hwlj krb5kdc[16854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:54:31.922083 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 17229
I20260926 20:54:31.922199 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/ts-2/wal/instance
I20260926 20:54:31.922408 392 external_mini_cluster.cc:1428] Setting key bdce6aa7cef53c7611f2d41e705fe349
I20260926 20:54:31.923523 16950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:31.914131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33045 (local address 127.0.98.62:40493)
0926 20:54:31.914178 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:31.914180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:31.914303 (+ 123us) server_negotiation.cc:407] Connection header received
0926 20:54:31.915228 (+ 925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:31.915229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:31.915239 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:31.915292 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:31.916000 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:31.916580 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:31.917743 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:31.917960 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:31.918033 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:31.920366 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:31.920367 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:31.920368 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:54:31.920374 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:31.921747 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:31.922329 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:31.922331 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:31.922331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:31.922387 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:31.922683 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:31.922684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:31.922684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:31.922875 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:31.922943 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:31.923313 (+ 370us) server_negotiation.cc:299] Negotiation successful
0926 20:54:31.923418 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 20:54:31.923718 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:31.913988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40493 (local address 127.0.98.3:33045)
0926 20:54:31.914165 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:31.914217 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:31.915140 (+ 923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:31.915353 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:31.915356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:31.915607 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:54:31.915879 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:31.915885 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:31.916707 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:31.916708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:31.917663 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:31.917665 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:31.918059 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:31.918060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:31.918177 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:31.918752 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:54:31.918763 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:54:31.920281 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:54:31.921830 (+ 1549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:54:31.921836 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:54:31.921838 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:54:31.922233 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:54:31.922458 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:54:31.922459 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:54:31.922460 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:54:31.922604 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:54:31.922963 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:54:31.922970 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:54:31.923161 (+ 191us) client_negotiation.cc:769] Sending connection context
0926 20:54:31.923278 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 20:54:31.923466 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":96,"threads_started":1}
I20260926 20:54:31.924175 17358 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40493
I20260926 20:54:31.924280 17358 heartbeater.cc:464] Registering TS with master...
I20260926 20:54:31.924467 17358 heartbeater.cc:511] Master 127.0.98.62:40493 requested a full tablet report, sending...
I20260926 20:54:31.924877 16898 ts_manager.cc:194] Registered new tserver with Master: a09035df5a744c43b508a66bc06c742f (127.0.98.3:39807)
I20260926 20:54:31.925264 16898 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33045
I20260926 20:54:31.934540 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:54:31.942085 16950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:31.936033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49190 (local address 127.0.98.62:40493)
0926 20:54:31.936153 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:31.936154 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:31.936242 (+ 88us) server_negotiation.cc:407] Connection header received
0926 20:54:31.936288 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:31.936288 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:31.936297 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:31.936352 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:31.937009 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:31.937500 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:31.938315 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:31.938502 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:31.938555 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:31.939809 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:31.939810 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:31.939810 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:54:31.939818 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:31.941080 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:31.941397 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:31.941398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:31.941398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:31.941448 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:31.941608 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:31.941608 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:31.941609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:31.941746 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:31.941818 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:31.941921 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 20:54:31.941979 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 20:54:31.943948 16898 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49190:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:54:31.945767 16898 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:54:31.952906 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:31.948230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33199 (local address 127.0.0.1:59614)
0926 20:54:31.948410 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:31.948451 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:31.948523 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:31.948969 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:31.948970 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:31.948971 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:54:31.949033 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:31.949036 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:31.950420 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:31.950421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:31.951827 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:31.951830 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:31.952569 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:31.952570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:31.952652 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:31.952667 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 20:54:31.952743 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 20:54:31.952800 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":8,"thread_start_us":63,"threads_started":1}
I20260926 20:54:31.952996 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:31.948754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39554 (local address 127.0.98.3:39807)
0926 20:54:31.949143 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:31.949148 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:31.949157 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:54:31.949196 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:31.949198 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:31.949212 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:31.949293 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:54:31.949473 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:31.950466 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:31.951926 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:31.952656 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:31.952720 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:31.952775 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:31.952868 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 20:54:31.952907 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":82,"threads_started":1}
I20260926 20:54:31.953027 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:31.948268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39807 (local address 127.0.0.1:39554)
0926 20:54:31.948639 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:31.948646 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:31.948676 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:31.949331 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:31.949333 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:31.949335 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:54:31.949389 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:31.949391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:31.950576 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:31.950577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:31.951827 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:31.951830 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:31.952730 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:31.952731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:31.952799 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:31.952802 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:54:31.952854 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 20:54:31.952888 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":9,"thread_start_us":96,"threads_started":1}
I20260926 20:54:31.953336 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:31.948331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59614 (local address 127.0.98.1:33199)
0926 20:54:31.948821 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:31.948826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:31.948836 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:54:31.948878 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:31.948880 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:31.948896 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:31.948974 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:54:31.949102 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:31.950283 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:31.951942 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:31.952468 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:31.952567 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:31.952637 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:31.953189 (+ 552us) server_negotiation.cc:299] Negotiation successful
0926 20:54:31.953231 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":84,"threads_started":1}
I20260926 20:54:31.954007 17024 tablet_service.cc:1467] Processing CreateTablet for tablet d33ac727a64a41649c73557741058abb (DEFAULT_TABLE table=test-table [id=2ade777aca134edd873f467ede5d06ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:54:31.953990 17292 tablet_service.cc:1467] Processing CreateTablet for tablet d33ac727a64a41649c73557741058abb (DEFAULT_TABLE table=test-table [id=2ade777aca134edd873f467ede5d06ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:54:31.954285 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:31.948520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47614 (local address 127.0.98.2:42279)
0926 20:54:31.948671 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:31.948676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:31.948687 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:54:31.948731 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:31.948733 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:31.948749 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:31.948837 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:54:31.950417 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:31.951328 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:31.952628 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:31.953896 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:31.953975 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:31.954033 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:31.954165 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 20:54:31.954209 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
I20260926 20:54:31.954378 17024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d33ac727a64a41649c73557741058abb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:54:31.954377 17292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d33ac727a64a41649c73557741058abb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:54:31.954489 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:31.948267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42279 (local address 127.0.0.1:47614)
0926 20:54:31.948494 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:31.948504 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:31.948546 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:31.948948 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:31.948952 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:31.948954 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:54:31.949033 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:31.949036 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:31.951450 (+ 2414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:31.951452 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:31.952521 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:31.952525 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:31.953988 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:31.953990 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:31.954058 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:31.954063 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:54:31.954125 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 20:54:31.954257 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":5,"thread_start_us":87,"threads_started":1}
I20260926 20:54:31.955168 17158 tablet_service.cc:1467] Processing CreateTablet for tablet d33ac727a64a41649c73557741058abb (DEFAULT_TABLE table=test-table [id=2ade777aca134edd873f467ede5d06ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:54:31.955493 17158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d33ac727a64a41649c73557741058abb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:54:31.957537 17375 tablet_bootstrap.cc:492] T d33ac727a64a41649c73557741058abb P a09035df5a744c43b508a66bc06c742f: Bootstrap starting.
I20260926 20:54:31.957935 17376 tablet_bootstrap.cc:492] T d33ac727a64a41649c73557741058abb P a594f0a4fde74a269009343cbec332d9: Bootstrap starting.
I20260926 20:54:31.958324 17375 tablet_bootstrap.cc:654] T d33ac727a64a41649c73557741058abb P a09035df5a744c43b508a66bc06c742f: Neither blocks nor log segments found. Creating new log.
I20260926 20:54:31.958587 17376 tablet_bootstrap.cc:654] T d33ac727a64a41649c73557741058abb P a594f0a4fde74a269009343cbec332d9: Neither blocks nor log segments found. Creating new log.
I20260926 20:54:31.958711 17377 tablet_bootstrap.cc:492] T d33ac727a64a41649c73557741058abb P 321a657e9e764eb5b9a192db8655c2eb: Bootstrap starting.
I20260926 20:54:31.959095 17375 log.cc:824] T d33ac727a64a41649c73557741058abb P a09035df5a744c43b508a66bc06c742f: Log is configured to *not* fsync() on all Append() calls
I20260926 20:54:31.959204 17376 log.cc:824] T d33ac727a64a41649c73557741058abb P a594f0a4fde74a269009343cbec332d9: Log is configured to *not* fsync() on all Append() calls
I20260926 20:54:31.959299 17377 tablet_bootstrap.cc:654] T d33ac727a64a41649c73557741058abb P 321a657e9e764eb5b9a192db8655c2eb: Neither blocks nor log segments found. Creating new log.
I20260926 20:54:31.959908 17377 log.cc:824] T d33ac727a64a41649c73557741058abb P 321a657e9e764eb5b9a192db8655c2eb: Log is configured to *not* fsync() on all Append() calls
I20260926 20:54:31.960042 17375 tablet_bootstrap.cc:492] T d33ac727a64a41649c73557741058abb P a09035df5a744c43b508a66bc06c742f: No bootstrap required, opened a new log
I20260926 20:54:31.960117 17375 ts_tablet_manager.cc:1397] T d33ac727a64a41649c73557741058abb P a09035df5a744c43b508a66bc06c742f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 20:54:31.960600 17377 tablet_bootstrap.cc:492] T d33ac727a64a41649c73557741058abb P 321a657e9e764eb5b9a192db8655c2eb: No bootstrap required, opened a new log
I20260926 20:54:31.960654 17377 ts_tablet_manager.cc:1397] T d33ac727a64a41649c73557741058abb P 321a657e9e764eb5b9a192db8655c2eb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:54:31.961591 17375 raft_consensus.cc:348] T d33ac727a64a41649c73557741058abb P a09035df5a744c43b508a66bc06c742f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a594f0a4fde74a269009343cbec332d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33199 } } peers { permanent_uuid: "a09035df5a744c43b508a66bc06c742f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39807 } } peers { permanent_uuid: "321a657e9e764eb5b9a192db8655c2eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42279 } }
I20260926 20:54:31.961766 17375 raft_consensus.cc:374] T d33ac727a64a41649c73557741058abb P a09035df5a744c43b508a66bc06c742f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:54:31.961815 17375 raft_consensus.cc:729] T d33ac727a64a41649c73557741058abb P a09035df5a744c43b508a66bc06c742f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a09035df5a744c43b508a66bc06c742f, State: Initialized, Role: FOLLOWER
I20260926 20:54:31.961928 17375 consensus_queue.cc:260] T d33ac727a64a41649c73557741058abb P a09035df5a744c43b508a66bc06c742f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a594f0a4fde74a269009343cbec332d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33199 } } peers { permanent_uuid: "a09035df5a744c43b508a66bc06c742f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39807 } } peers { permanent_uuid: "321a657e9e764eb5b9a192db8655c2eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42279 } }
I20260926 20:54:31.962065 17377 raft_consensus.cc:348] T d33ac727a64a41649c73557741058abb P 321a657e9e764eb5b9a192db8655c2eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a594f0a4fde74a269009343cbec332d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33199 } } peers { permanent_uuid: "a09035df5a744c43b508a66bc06c742f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39807 } } peers { permanent_uuid: "321a657e9e764eb5b9a192db8655c2eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42279 } }
I20260926 20:54:31.962240 17377 raft_consensus.cc:374] T d33ac727a64a41649c73557741058abb P 321a657e9e764eb5b9a192db8655c2eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:54:31.962252 17375 ts_tablet_manager.cc:1428] T d33ac727a64a41649c73557741058abb P a09035df5a744c43b508a66bc06c742f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:54:31.962255 17358 heartbeater.cc:503] Master 127.0.98.62:40493 was elected leader, sending a full tablet report...
I20260926 20:54:31.962284 17377 raft_consensus.cc:729] T d33ac727a64a41649c73557741058abb P 321a657e9e764eb5b9a192db8655c2eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 321a657e9e764eb5b9a192db8655c2eb, State: Initialized, Role: FOLLOWER
I20260926 20:54:31.962392 17377 consensus_queue.cc:260] T d33ac727a64a41649c73557741058abb P 321a657e9e764eb5b9a192db8655c2eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a594f0a4fde74a269009343cbec332d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33199 } } peers { permanent_uuid: "a09035df5a744c43b508a66bc06c742f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39807 } } peers { permanent_uuid: "321a657e9e764eb5b9a192db8655c2eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42279 } }
I20260926 20:54:31.962666 17377 ts_tablet_manager.cc:1428] T d33ac727a64a41649c73557741058abb P 321a657e9e764eb5b9a192db8655c2eb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:54:31.962684 17224 heartbeater.cc:503] Master 127.0.98.62:40493 was elected leader, sending a full tablet report...
I20260926 20:54:31.962757 17376 tablet_bootstrap.cc:492] T d33ac727a64a41649c73557741058abb P a594f0a4fde74a269009343cbec332d9: No bootstrap required, opened a new log
I20260926 20:54:31.962819 17376 ts_tablet_manager.cc:1397] T d33ac727a64a41649c73557741058abb P a594f0a4fde74a269009343cbec332d9: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260926 20:54:31.964715 17376 raft_consensus.cc:348] T d33ac727a64a41649c73557741058abb P a594f0a4fde74a269009343cbec332d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a594f0a4fde74a269009343cbec332d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33199 } } peers { permanent_uuid: "a09035df5a744c43b508a66bc06c742f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39807 } } peers { permanent_uuid: "321a657e9e764eb5b9a192db8655c2eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42279 } }
I20260926 20:54:31.964857 17376 raft_consensus.cc:374] T d33ac727a64a41649c73557741058abb P a594f0a4fde74a269009343cbec332d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:54:31.964880 17376 raft_consensus.cc:729] T d33ac727a64a41649c73557741058abb P a594f0a4fde74a269009343cbec332d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a594f0a4fde74a269009343cbec332d9, State: Initialized, Role: FOLLOWER
I20260926 20:54:31.964989 17376 consensus_queue.cc:260] T d33ac727a64a41649c73557741058abb P a594f0a4fde74a269009343cbec332d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a594f0a4fde74a269009343cbec332d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33199 } } peers { permanent_uuid: "a09035df5a744c43b508a66bc06c742f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39807 } } peers { permanent_uuid: "321a657e9e764eb5b9a192db8655c2eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42279 } }
I20260926 20:54:31.965261 17376 ts_tablet_manager.cc:1428] T d33ac727a64a41649c73557741058abb P a594f0a4fde74a269009343cbec332d9: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 20:54:31.965315 17090 heartbeater.cc:503] Master 127.0.98.62:40493 was elected leader, sending a full tablet report...
I20260926 20:54:31.986340 17381 raft_consensus.cc:482] T d33ac727a64a41649c73557741058abb P a09035df5a744c43b508a66bc06c742f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:54:31.986503 17381 raft_consensus.cc:504] T d33ac727a64a41649c73557741058abb P a09035df5a744c43b508a66bc06c742f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a594f0a4fde74a269009343cbec332d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33199 } } peers { permanent_uuid: "a09035df5a744c43b508a66bc06c742f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39807 } } peers { permanent_uuid: "321a657e9e764eb5b9a192db8655c2eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42279 } }
I20260926 20:54:31.986937 17381 leader_election.cc:288] T d33ac727a64a41649c73557741058abb P a09035df5a744c43b508a66bc06c742f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a594f0a4fde74a269009343cbec332d9 (127.0.98.1:33199), 321a657e9e764eb5b9a192db8655c2eb (127.0.98.2:42279)
I20260926 20:54:31.990882 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:31.987174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42465 (local address 127.0.98.2:42279)
0926 20:54:31.987221 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:31.987223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:31.987233 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:54:31.987261 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:31.987261 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:31.987272 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:31.987337 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:54:31.987564 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:31.988374 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:31.989950 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:31.990521 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:31.990597 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:31.990608 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:31.990768 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 20:54:31.990809 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 20:54:31.990882 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:31.987258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45431 (local address 127.0.98.1:33199)
0926 20:54:31.987311 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:31.987313 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:31.987406 (+ 93us) server_negotiation.cc:407] Connection header received
0926 20:54:31.987453 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:31.987453 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:31.987463 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:31.987504 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:54:31.987659 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:31.988369 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:31.989950 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:31.990531 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:31.990597 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:31.990608 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:31.990768 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 20:54:31.990812 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 20:54:31.990904 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:31.987097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42279 (local address 127.0.98.3:42465)
0926 20:54:31.987135 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:31.987149 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:31.987196 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:31.987379 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:31.987380 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:31.987382 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:54:31.987459 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:31.987460 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:31.988458 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:31.988459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:31.989855 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:31.989858 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:31.990608 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:31.990609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:31.990688 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:31.990692 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:54:31.990750 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 20:54:31.990799 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19,"mutex_wait_us":115}
I20260926 20:54:31.990922 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:31.987112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33199 (local address 127.0.98.3:45431)
0926 20:54:31.987299 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:31.987335 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:31.987393 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:31.987531 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:31.987532 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:31.987533 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:54:31.987597 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:31.987598 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:31.988458 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:31.988459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:31.989855 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:31.989858 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:31.990608 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:31.990609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:31.990688 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:31.990692 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:54:31.990750 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 20:54:31.990799 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":77,"thread_start_us":117,"threads_started":1}
I20260926 20:54:31.991289 17178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d33ac727a64a41649c73557741058abb" candidate_uuid: "a09035df5a744c43b508a66bc06c742f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "321a657e9e764eb5b9a192db8655c2eb" is_pre_election: true
I20260926 20:54:31.991488 17178 raft_consensus.cc:2454] T d33ac727a64a41649c73557741058abb P 321a657e9e764eb5b9a192db8655c2eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a09035df5a744c43b508a66bc06c742f in term 0.
I20260926 20:54:31.991818 17044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d33ac727a64a41649c73557741058abb" candidate_uuid: "a09035df5a744c43b508a66bc06c742f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a594f0a4fde74a269009343cbec332d9" is_pre_election: true
I20260926 20:54:31.991926 17246 leader_election.cc:302] T d33ac727a64a41649c73557741058abb P a09035df5a744c43b508a66bc06c742f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 321a657e9e764eb5b9a192db8655c2eb, a09035df5a744c43b508a66bc06c742f; no voters:
I20260926 20:54:31.991973 17044 raft_consensus.cc:2454] T d33ac727a64a41649c73557741058abb P a594f0a4fde74a269009343cbec332d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a09035df5a744c43b508a66bc06c742f in term 0.
I20260926 20:54:31.992058 17381 raft_consensus.cc:2781] T d33ac727a64a41649c73557741058abb P a09035df5a744c43b508a66bc06c742f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:54:31.992115 17381 raft_consensus.cc:482] T d33ac727a64a41649c73557741058abb P a09035df5a744c43b508a66bc06c742f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:54:31.992138 17381 raft_consensus.cc:3037] T d33ac727a64a41649c73557741058abb P a09035df5a744c43b508a66bc06c742f [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:54:31.992864 17381 raft_consensus.cc:504] T d33ac727a64a41649c73557741058abb P a09035df5a744c43b508a66bc06c742f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a594f0a4fde74a269009343cbec332d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33199 } } peers { permanent_uuid: "a09035df5a744c43b508a66bc06c742f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39807 } } peers { permanent_uuid: "321a657e9e764eb5b9a192db8655c2eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42279 } }
I20260926 20:54:31.993005 17381 leader_election.cc:288] T d33ac727a64a41649c73557741058abb P a09035df5a744c43b508a66bc06c742f [CANDIDATE]: Term 1 election: Requested vote from peers a594f0a4fde74a269009343cbec332d9 (127.0.98.1:33199), 321a657e9e764eb5b9a192db8655c2eb (127.0.98.2:42279)
I20260926 20:54:31.993170 17044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d33ac727a64a41649c73557741058abb" candidate_uuid: "a09035df5a744c43b508a66bc06c742f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a594f0a4fde74a269009343cbec332d9"
I20260926 20:54:31.993191 17178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d33ac727a64a41649c73557741058abb" candidate_uuid: "a09035df5a744c43b508a66bc06c742f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "321a657e9e764eb5b9a192db8655c2eb"
I20260926 20:54:31.993256 17178 raft_consensus.cc:3037] T d33ac727a64a41649c73557741058abb P 321a657e9e764eb5b9a192db8655c2eb [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:54:31.993252 17044 raft_consensus.cc:3037] T d33ac727a64a41649c73557741058abb P a594f0a4fde74a269009343cbec332d9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:54:31.994119 17044 raft_consensus.cc:2454] T d33ac727a64a41649c73557741058abb P a594f0a4fde74a269009343cbec332d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a09035df5a744c43b508a66bc06c742f in term 1.
I20260926 20:54:31.994150 17178 raft_consensus.cc:2454] T d33ac727a64a41649c73557741058abb P 321a657e9e764eb5b9a192db8655c2eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a09035df5a744c43b508a66bc06c742f in term 1.
I20260926 20:54:31.994273 17247 leader_election.cc:302] T d33ac727a64a41649c73557741058abb P a09035df5a744c43b508a66bc06c742f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a09035df5a744c43b508a66bc06c742f, a594f0a4fde74a269009343cbec332d9; no voters:
I20260926 20:54:31.994386 17381 raft_consensus.cc:2781] T d33ac727a64a41649c73557741058abb P a09035df5a744c43b508a66bc06c742f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:54:31.994524 17381 raft_consensus.cc:686] T d33ac727a64a41649c73557741058abb P a09035df5a744c43b508a66bc06c742f [term 1 LEADER]: Becoming Leader. State: Replica: a09035df5a744c43b508a66bc06c742f, State: Running, Role: LEADER
I20260926 20:54:31.994627 17381 consensus_queue.cc:237] T d33ac727a64a41649c73557741058abb P a09035df5a744c43b508a66bc06c742f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a594f0a4fde74a269009343cbec332d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33199 } } peers { permanent_uuid: "a09035df5a744c43b508a66bc06c742f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39807 } } peers { permanent_uuid: "321a657e9e764eb5b9a192db8655c2eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42279 } }
I20260926 20:54:31.995460 16899 catalog_manager.cc:5495] T d33ac727a64a41649c73557741058abb P a09035df5a744c43b508a66bc06c742f reported cstate change: term changed from 0 to 1, leader changed from <none> to a09035df5a744c43b508a66bc06c742f (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a09035df5a744c43b508a66bc06c742f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a594f0a4fde74a269009343cbec332d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a09035df5a744c43b508a66bc06c742f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "321a657e9e764eb5b9a192db8655c2eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42279 } health_report { overall_health: UNKNOWN } } }
W20260926 20:54:31.999889 17225 tablet.cc:2369] T d33ac727a64a41649c73557741058abb P 321a657e9e764eb5b9a192db8655c2eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:54:32.010500 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:32.008197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39566 (local address 127.0.98.3:39807)
0926 20:54:32.008260 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:32.008262 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:32.008275 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:54:32.008309 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:32.008310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:32.008326 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:32.008388 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:54:32.008498 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:32.009012 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:32.009763 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:32.009973 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:32.010047 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:32.010186 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:54:32.010292 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:54:32.010361 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:32.010396 (+ 35us) server_negotiation.cc:299] Negotiation successful
0926 20:54:32.010418 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 20:54:32.012408 17044 raft_consensus.cc:1261] T d33ac727a64a41649c73557741058abb P a594f0a4fde74a269009343cbec332d9 [term 1 FOLLOWER]: Refusing update from remote peer a09035df5a744c43b508a66bc06c742f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:54:32.012418 17178 raft_consensus.cc:1261] T d33ac727a64a41649c73557741058abb P 321a657e9e764eb5b9a192db8655c2eb [term 1 FOLLOWER]: Refusing update from remote peer a09035df5a744c43b508a66bc06c742f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:54:32.012784 17385 consensus_queue.cc:1035] T d33ac727a64a41649c73557741058abb P a09035df5a744c43b508a66bc06c742f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a594f0a4fde74a269009343cbec332d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:54:32.012943 17381 consensus_queue.cc:1035] T d33ac727a64a41649c73557741058abb P a09035df5a744c43b508a66bc06c742f [LEADER]: Connected to new peer: Peer: permanent_uuid: "321a657e9e764eb5b9a192db8655c2eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:54:32.015287 17388 mvcc.cc:205] Tried to move back new op lower bound from 7333708070959976448 to 7333708070890713088. Current Snapshot: MvccSnapshot[applied={T|T < 7333708070959976448}]
I20260926 20:54:32.016251 17389 mvcc.cc:205] Tried to move back new op lower bound from 7333708070959976448 to 7333708070890713088. Current Snapshot: MvccSnapshot[applied={T|T < 7333708070959976448}]
I20260926 20:54:32.017735 16899 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:54:32.017835 16899 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:54:32.018630 16899 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bb9301410c9c45788c3011fd4f40df08: Sending DeleteTablet for 3 replicas of tablet d33ac727a64a41649c73557741058abb
I20260926 20:54:32.018972 17292 tablet_service.cc:1514] Processing DeleteTablet for tablet d33ac727a64a41649c73557741058abb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:54:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39554
I20260926 20:54:32.018996 17158 tablet_service.cc:1514] Processing DeleteTablet for tablet d33ac727a64a41649c73557741058abb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:54:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47614
I20260926 20:54:32.019071 17024 tablet_service.cc:1514] Processing DeleteTablet for tablet d33ac727a64a41649c73557741058abb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:54:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59614
I20260926 20:54:32.019236 17399 tablet_replica.cc:330] T d33ac727a64a41649c73557741058abb P 321a657e9e764eb5b9a192db8655c2eb: stopping tablet replica
I20260926 20:54:32.019349 17399 raft_consensus.cc:2229] T d33ac727a64a41649c73557741058abb P 321a657e9e764eb5b9a192db8655c2eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:54:32.019451 17399 raft_consensus.cc:2258] T d33ac727a64a41649c73557741058abb P 321a657e9e764eb5b9a192db8655c2eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:54:32.019585 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 16961
I20260926 20:54:32.020287 17399 ts_tablet_manager.cc:1905] T d33ac727a64a41649c73557741058abb P 321a657e9e764eb5b9a192db8655c2eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:54:32.021876 17399 ts_tablet_manager.cc:1918] T d33ac727a64a41649c73557741058abb P 321a657e9e764eb5b9a192db8655c2eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:54:32.021935 17399 log.cc:1196] T d33ac727a64a41649c73557741058abb P 321a657e9e764eb5b9a192db8655c2eb: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/ts-1/wal/wals/d33ac727a64a41649c73557741058abb
I20260926 20:54:32.022218 17399 ts_tablet_manager.cc:1939] T d33ac727a64a41649c73557741058abb P 321a657e9e764eb5b9a192db8655c2eb: Deleting consensus metadata
I20260926 20:54:32.022652 16884 catalog_manager.cc:4841] TS 321a657e9e764eb5b9a192db8655c2eb (127.0.98.2:42279): tablet d33ac727a64a41649c73557741058abb (table test-table [id=2ade777aca134edd873f467ede5d06ee]) successfully deleted
I20260926 20:54:32.023895 17398 tablet_replica.cc:330] T d33ac727a64a41649c73557741058abb P a09035df5a744c43b508a66bc06c742f: stopping tablet replica
I20260926 20:54:32.024030 17398 raft_consensus.cc:2229] T d33ac727a64a41649c73557741058abb P a09035df5a744c43b508a66bc06c742f [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:54:32.024137 17398 raft_consensus.cc:2258] T d33ac727a64a41649c73557741058abb P a09035df5a744c43b508a66bc06c742f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:54:32.024536 17398 ts_tablet_manager.cc:1905] T d33ac727a64a41649c73557741058abb P a09035df5a744c43b508a66bc06c742f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 20:54:32.025615 16885 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33199 (error 108)
I20260926 20:54:32.025874 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 17095
I20260926 20:54:32.026494 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:32.025986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33199 (local address 127.0.0.1:59630)
0926 20:54:32.026415 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:32.026452 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407}
I20260926 20:54:32.026517 17398 ts_tablet_manager.cc:1918] T d33ac727a64a41649c73557741058abb P a09035df5a744c43b508a66bc06c742f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:54:32.026584 17398 log.cc:1196] T d33ac727a64a41649c73557741058abb P a09035df5a744c43b508a66bc06c742f: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455725523895-392-0/minicluster-data/ts-2/wal/wals/d33ac727a64a41649c73557741058abb
I20260926 20:54:32.026847 17398 ts_tablet_manager.cc:1939] T d33ac727a64a41649c73557741058abb P a09035df5a744c43b508a66bc06c742f: Deleting consensus metadata
I20260926 20:54:32.027273 16883 catalog_manager.cc:4841] TS a09035df5a744c43b508a66bc06c742f (127.0.98.3:39807): tablet d33ac727a64a41649c73557741058abb (table test-table [id=2ade777aca134edd873f467ede5d06ee]) successfully deleted
W20260926 20:54:32.028311 16885 catalog_manager.cc:4584] TS a594f0a4fde74a269009343cbec332d9 (127.0.98.1:33199): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d33ac727a64a41649c73557741058abb: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33199: connect: Connection refused (error 111)
I20260926 20:54:32.031961 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 17229
I20260926 20:54:32.037120 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 16867
2026-09-26T20:54:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1942 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T20:54:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:54:32Z Disabled control of system clock
I20260926 20:54:32.064466 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40713
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40713
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 20:54:32.147990 17405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:54:32.148212 17405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:54:32.148233 17405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:54:32.149542 17405 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:54:32.149590 17405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:54:32.149603 17405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:54:32.149645 17405 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:54:32.149674 17405 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:54:32.151077 17405 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40713
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:54:32.151355 17405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:54:32.151669 17405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:54:32.156544 17405 server_base.cc:1034] running on GCE node
W20260926 20:54:32.156528 17413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:32.156509 17411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:32.156517 17410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:32.156934 17405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:54:32.157236 17405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:54:32.158382 17405 hybrid_clock.cc:648] HybridClock initialized: now 1790456072158361 us; error 55 us; skew 500 ppm
I20260926 20:54:32.159834 17405 webserver.cc:460] Webserver started at http://127.0.98.62:45491/ using document root <none> and password file <none>
I20260926 20:54:32.160125 17405 fs_manager.cc:359] Metadata directory not provided
I20260926 20:54:32.160194 17405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:54:32.160367 17405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:54:32.161556 17405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/master-0/data/instance:
uuid: "9dc7c9c1a36a496cb79887de8b4fb431"
format_stamp: "Formatted at 2026-09-26 20:54:32 on dist-test-slave-hwlj"
server_key: "683e0e9b942fd83022ac6abc720a48c7"
server_key_iv: "dc3c4c714ef89440b7866dcd52150e04"
server_key_version: "encryptionkey@0"
I20260926 20:54:32.161931 17405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/master-0/wal/instance:
uuid: "9dc7c9c1a36a496cb79887de8b4fb431"
format_stamp: "Formatted at 2026-09-26 20:54:32 on dist-test-slave-hwlj"
server_key: "683e0e9b942fd83022ac6abc720a48c7"
server_key_iv: "dc3c4c714ef89440b7866dcd52150e04"
server_key_version: "encryptionkey@0"
I20260926 20:54:32.163316 17405 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:54:32.164304 17419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:32.164548 17405 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 20:54:32.164690 17405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "9dc7c9c1a36a496cb79887de8b4fb431"
format_stamp: "Formatted at 2026-09-26 20:54:32 on dist-test-slave-hwlj"
server_key: "683e0e9b942fd83022ac6abc720a48c7"
server_key_iv: "dc3c4c714ef89440b7866dcd52150e04"
server_key_version: "encryptionkey@0"
I20260926 20:54:32.164772 17405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:54:32.174490 17405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:54:32.174822 17405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:54:32.174980 17405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:54:32.179255 17405 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40713
I20260926 20:54:32.179324 17471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40713 every 8 connection(s)
I20260926 20:54:32.179694 17405 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:54:32.180732 17472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:54:32.183116 17472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dc7c9c1a36a496cb79887de8b4fb431: Bootstrap starting.
I20260926 20:54:32.183714 17472 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9dc7c9c1a36a496cb79887de8b4fb431: Neither blocks nor log segments found. Creating new log.
I20260926 20:54:32.184059 17472 log.cc:824] T 00000000000000000000000000000000 P 9dc7c9c1a36a496cb79887de8b4fb431: Log is configured to *not* fsync() on all Append() calls
I20260926 20:54:32.184767 17472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dc7c9c1a36a496cb79887de8b4fb431: No bootstrap required, opened a new log
I20260926 20:54:32.186378 17472 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9dc7c9c1a36a496cb79887de8b4fb431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dc7c9c1a36a496cb79887de8b4fb431" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40713 } }
I20260926 20:54:32.186515 17472 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9dc7c9c1a36a496cb79887de8b4fb431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:54:32.186538 17472 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9dc7c9c1a36a496cb79887de8b4fb431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dc7c9c1a36a496cb79887de8b4fb431, State: Initialized, Role: FOLLOWER
I20260926 20:54:32.186710 17472 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9dc7c9c1a36a496cb79887de8b4fb431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dc7c9c1a36a496cb79887de8b4fb431" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40713 } }
I20260926 20:54:32.186796 17472 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9dc7c9c1a36a496cb79887de8b4fb431 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:54:32.186851 17472 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9dc7c9c1a36a496cb79887de8b4fb431 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:54:32.186903 17472 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9dc7c9c1a36a496cb79887de8b4fb431 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:54:32.187441 17472 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9dc7c9c1a36a496cb79887de8b4fb431 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dc7c9c1a36a496cb79887de8b4fb431" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40713 } }
I20260926 20:54:32.187551 17472 leader_election.cc:302] T 00000000000000000000000000000000 P 9dc7c9c1a36a496cb79887de8b4fb431 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9dc7c9c1a36a496cb79887de8b4fb431; no voters:
I20260926 20:54:32.187726 17472 leader_election.cc:288] T 00000000000000000000000000000000 P 9dc7c9c1a36a496cb79887de8b4fb431 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:54:32.187844 17475 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9dc7c9c1a36a496cb79887de8b4fb431 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:54:32.188009 17475 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9dc7c9c1a36a496cb79887de8b4fb431 [term 1 LEADER]: Becoming Leader. State: Replica: 9dc7c9c1a36a496cb79887de8b4fb431, State: Running, Role: LEADER
I20260926 20:54:32.188027 17472 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9dc7c9c1a36a496cb79887de8b4fb431 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:54:32.188103 17475 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9dc7c9c1a36a496cb79887de8b4fb431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dc7c9c1a36a496cb79887de8b4fb431" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40713 } }
I20260926 20:54:32.188544 17477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dc7c9c1a36a496cb79887de8b4fb431 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dc7c9c1a36a496cb79887de8b4fb431. Latest consensus state: current_term: 1 leader_uuid: "9dc7c9c1a36a496cb79887de8b4fb431" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dc7c9c1a36a496cb79887de8b4fb431" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40713 } } }
I20260926 20:54:32.188623 17477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dc7c9c1a36a496cb79887de8b4fb431 [sys.catalog]: This master's current role is: LEADER
I20260926 20:54:32.188643 17476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dc7c9c1a36a496cb79887de8b4fb431 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9dc7c9c1a36a496cb79887de8b4fb431" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dc7c9c1a36a496cb79887de8b4fb431" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40713 } } }
I20260926 20:54:32.188740 17476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dc7c9c1a36a496cb79887de8b4fb431 [sys.catalog]: This master's current role is: LEADER
I20260926 20:54:32.188863 17484 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:54:32.189273 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 17405
I20260926 20:54:32.189356 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/master-0/wal/instance
I20260926 20:54:32.189561 392 external_mini_cluster.cc:1428] Setting key 421424b1be05f21a08864096582062ed
I20260926 20:54:32.190119 17484 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:54:32.191900 17484 catalog_manager.cc:1347] Generated new cluster ID: b931f3df7e724f6aa5a520c92cf95a40
I20260926 20:54:32.191951 17484 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:54:32.196556 17491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:32.190097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59126 (local address 127.0.98.62:40713)
0926 20:54:32.190459 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:32.190467 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:32.190497 (+ 30us) server_negotiation.cc:407] Connection header received
0926 20:54:32.190803 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:32.190811 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:32.190887 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:32.191274 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:32.191488 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:32.192912 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:32.194080 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:32.194709 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:32.195181 (+ 472us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:32.195590 (+ 409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:32.195594 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:32.195594 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:54:32.195855 (+ 261us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:32.195896 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:54:32.195911 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:32.196215 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:32.196250 (+ 35us) server_negotiation.cc:299] Negotiation successful
0926 20:54:32.196335 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":72,"threads_started":1}
I20260926 20:54:32.206212 17484 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:54:32.206755 17484 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:54:32.208712 17484 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9dc7c9c1a36a496cb79887de8b4fb431: Generated new TSK 0
I20260926 20:54:32.208896 17484 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:54:32.210958 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40713
--builtin_ntp_servers=127.0.98.20:46059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 20:54:32.296684 17496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:54:32.296898 17496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:54:32.296919 17496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:54:32.298094 17496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:54:32.298137 17496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:54:32.298202 17496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:54:32.299578 17496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:54:32.299883 17496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:54:32.300179 17496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:54:32.306196 17502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:32.306202 17501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:32.306447 17496 server_base.cc:1034] running on GCE node
W20260926 20:54:32.306598 17504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:32.306826 17496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:54:32.307081 17496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:54:32.308251 17496 hybrid_clock.cc:648] HybridClock initialized: now 1790456072308237 us; error 32 us; skew 500 ppm
I20260926 20:54:32.309582 17496 webserver.cc:460] Webserver started at http://127.0.98.1:46513/ using document root <none> and password file <none>
I20260926 20:54:32.309849 17496 fs_manager.cc:359] Metadata directory not provided
I20260926 20:54:32.309919 17496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:54:32.310062 17496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:54:32.311167 17496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/ts-0/data/instance:
uuid: "8c1618f4e070465aba788ee84db549ed"
format_stamp: "Formatted at 2026-09-26 20:54:32 on dist-test-slave-hwlj"
server_key: "4a025f9866a1dec8485c4bf945040b9b"
server_key_iv: "40edd4e032803045068d1071ec9dc1b4"
server_key_version: "encryptionkey@0"
I20260926 20:54:32.311517 17496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8c1618f4e070465aba788ee84db549ed"
format_stamp: "Formatted at 2026-09-26 20:54:32 on dist-test-slave-hwlj"
server_key: "4a025f9866a1dec8485c4bf945040b9b"
server_key_iv: "40edd4e032803045068d1071ec9dc1b4"
server_key_version: "encryptionkey@0"
I20260926 20:54:32.312930 17496 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:54:32.313740 17510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:32.313910 17496 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:54:32.313988 17496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/ts-0/wal
uuid: "8c1618f4e070465aba788ee84db549ed"
format_stamp: "Formatted at 2026-09-26 20:54:32 on dist-test-slave-hwlj"
server_key: "4a025f9866a1dec8485c4bf945040b9b"
server_key_iv: "40edd4e032803045068d1071ec9dc1b4"
server_key_version: "encryptionkey@0"
I20260926 20:54:32.314076 17496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:54:32.321388 17496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:54:32.321671 17496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:54:32.321817 17496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:54:32.322038 17496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:54:32.322340 17496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:54:32.322371 17496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:32.322424 17496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:54:32.322451 17496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:32.328428 17496 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38071
I20260926 20:54:32.328540 17623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38071 every 8 connection(s)
I20260926 20:54:32.328846 17496 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:54:32.334125 17491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:32.329616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56375 (local address 127.0.98.62:40713)
0926 20:54:32.329680 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:32.329681 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:32.329979 (+ 298us) server_negotiation.cc:407] Connection header received
0926 20:54:32.330433 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:32.330434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:32.330446 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:32.330498 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:32.331148 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:32.331634 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:32.332756 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:32.332971 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:32.333038 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:32.333871 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:32.333873 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:32.333873 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:54:32.333881 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:32.333897 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:54:32.333901 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:32.333948 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:32.334039 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 20:54:32.334064 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 20:54:32.334299 17626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:32.329617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40713 (local address 127.0.98.1:56375)
0926 20:54:32.329852 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:32.329886 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:32.330354 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:32.330560 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:32.330563 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:32.330572 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:54:32.331069 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:32.331073 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:32.331716 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:32.331717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:32.332620 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:32.332623 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:32.333063 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:32.333064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:32.333180 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:32.333688 (+ 508us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:54:32.333688 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:54:32.333720 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:54:32.333743 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:54:32.333959 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:54:32.333963 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:54:32.333966 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:54:32.334018 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 20:54:32.334097 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":101,"threads_started":1}
I20260926 20:54:32.334815 17624 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40713
I20260926 20:54:32.334934 17624 heartbeater.cc:464] Registering TS with master...
I20260926 20:54:32.335120 17624 heartbeater.cc:511] Master 127.0.98.62:40713 requested a full tablet report, sending...
I20260926 20:54:32.335626 17436 ts_manager.cc:194] Registered new tserver with Master: 8c1618f4e070465aba788ee84db549ed (127.0.98.1:38071)
I20260926 20:54:32.336256 17436 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:56375
I20260926 20:54:32.338856 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 17496
I20260926 20:54:32.338940 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/ts-0/wal/instance
I20260926 20:54:32.339093 392 external_mini_cluster.cc:1428] Setting key 602875b24c8bf4e2627661d36f2e21b1
I20260926 20:54:32.340046 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40713
--builtin_ntp_servers=127.0.98.20:46059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 20:54:32.423101 17627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:54:32.423313 17627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:54:32.423336 17627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:54:32.424535 17627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:54:32.424579 17627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:54:32.424610 17627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:54:32.426313 17627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:54:32.426512 17627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:54:32.426752 17627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:54:32.431661 17635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:32.431767 17633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:32.431874 17632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:32.431828 17627 server_base.cc:1034] running on GCE node
I20260926 20:54:32.432125 17627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:54:32.432356 17627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:54:32.433504 17627 hybrid_clock.cc:648] HybridClock initialized: now 1790456072433483 us; error 37 us; skew 500 ppm
I20260926 20:54:32.434850 17627 webserver.cc:460] Webserver started at http://127.0.98.2:32775/ using document root <none> and password file <none>
I20260926 20:54:32.435082 17627 fs_manager.cc:359] Metadata directory not provided
I20260926 20:54:32.435127 17627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:54:32.435292 17627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:54:32.436447 17627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/ts-1/data/instance:
uuid: "8c869237bb14499982b2dd56e9b463c7"
format_stamp: "Formatted at 2026-09-26 20:54:32 on dist-test-slave-hwlj"
server_key: "1c9778534ea96eb43665cb7d8874c8f1"
server_key_iv: "d3f8d56bbe7fd7d5d5f1755bfb4cc82b"
server_key_version: "encryptionkey@0"
I20260926 20:54:32.436792 17627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8c869237bb14499982b2dd56e9b463c7"
format_stamp: "Formatted at 2026-09-26 20:54:32 on dist-test-slave-hwlj"
server_key: "1c9778534ea96eb43665cb7d8874c8f1"
server_key_iv: "d3f8d56bbe7fd7d5d5f1755bfb4cc82b"
server_key_version: "encryptionkey@0"
I20260926 20:54:32.438097 17627 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:54:32.438815 17641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:32.438997 17627 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:54:32.439085 17627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/ts-1/wal
uuid: "8c869237bb14499982b2dd56e9b463c7"
format_stamp: "Formatted at 2026-09-26 20:54:32 on dist-test-slave-hwlj"
server_key: "1c9778534ea96eb43665cb7d8874c8f1"
server_key_iv: "d3f8d56bbe7fd7d5d5f1755bfb4cc82b"
server_key_version: "encryptionkey@0"
I20260926 20:54:32.439152 17627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:54:32.450172 17627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:54:32.450481 17627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:54:32.450621 17627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:54:32.450860 17627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:54:32.451210 17627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:54:32.451262 17627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:32.451306 17627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:54:32.451334 17627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:32.457523 17627 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45635
I20260926 20:54:32.457571 17754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45635 every 8 connection(s)
I20260926 20:54:32.457908 17627 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
I20260926 20:54:32.463003 17491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:32.458625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48017 (local address 127.0.98.62:40713)
0926 20:54:32.458688 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:32.458690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:32.458946 (+ 256us) server_negotiation.cc:407] Connection header received
0926 20:54:32.459390 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:32.459392 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:32.459403 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:32.459467 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:32.460211 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:32.460690 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:32.461708 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:32.461893 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:32.461947 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:32.462773 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:32.462774 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:32.462774 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:54:32.462778 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:32.462793 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:54:32.462796 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:32.462840 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:32.462921 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 20:54:32.462944 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 20:54:32.463227 17757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:32.458621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40713 (local address 127.0.98.2:48017)
0926 20:54:32.458797 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:32.458850 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:32.459306 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:32.459520 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:32.459523 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:32.459531 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:54:32.460115 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:32.460120 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:32.460802 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:32.460803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:32.461608 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:32.461610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:32.461977 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:32.461978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:32.462094 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:32.462590 (+ 496us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:54:32.462591 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:54:32.462618 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:54:32.462640 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:54:32.462852 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:54:32.462856 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:54:32.462858 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 20:54:32.462910 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 20:54:32.462989 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":75,"threads_started":1}
I20260926 20:54:32.463685 17755 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40713
I20260926 20:54:32.463817 17755 heartbeater.cc:464] Registering TS with master...
I20260926 20:54:32.464000 17755 heartbeater.cc:511] Master 127.0.98.62:40713 requested a full tablet report, sending...
I20260926 20:54:32.464386 17436 ts_manager.cc:194] Registered new tserver with Master: 8c869237bb14499982b2dd56e9b463c7 (127.0.98.2:45635)
I20260926 20:54:32.464674 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 17627
I20260926 20:54:32.464771 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/ts-1/wal/instance
I20260926 20:54:32.464915 17436 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:48017
I20260926 20:54:32.465008 392 external_mini_cluster.cc:1428] Setting key 36bd52796483449e1c4fe157a25ee2db
I20260926 20:54:32.465941 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40713
--builtin_ntp_servers=127.0.98.20:46059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 20:54:32.546497 17758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:54:32.546753 17758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:54:32.546792 17758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:54:32.547986 17758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:54:32.548049 17758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:54:32.548090 17758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:54:32.549464 17758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:54:32.549731 17758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:54:32.550033 17758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:54:32.555051 17763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:32.555106 17766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:32.555114 17764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:32.555312 17758 server_base.cc:1034] running on GCE node
I20260926 20:54:32.555548 17758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:54:32.555824 17758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:54:32.556977 17758 hybrid_clock.cc:648] HybridClock initialized: now 1790456072556961 us; error 33 us; skew 500 ppm
I20260926 20:54:32.558270 17758 webserver.cc:460] Webserver started at http://127.0.98.3:36365/ using document root <none> and password file <none>
I20260926 20:54:32.558521 17758 fs_manager.cc:359] Metadata directory not provided
I20260926 20:54:32.558589 17758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:54:32.558736 17758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:54:32.559888 17758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/ts-2/data/instance:
uuid: "d077c61ed5564832b616aa143e2f1894"
format_stamp: "Formatted at 2026-09-26 20:54:32 on dist-test-slave-hwlj"
server_key: "1cecd0b1d3697e2b43c44decebc0390c"
server_key_iv: "6d5af3aadfcd69d06bdd8d11e596ccde"
server_key_version: "encryptionkey@0"
I20260926 20:54:32.560241 17758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d077c61ed5564832b616aa143e2f1894"
format_stamp: "Formatted at 2026-09-26 20:54:32 on dist-test-slave-hwlj"
server_key: "1cecd0b1d3697e2b43c44decebc0390c"
server_key_iv: "6d5af3aadfcd69d06bdd8d11e596ccde"
server_key_version: "encryptionkey@0"
I20260926 20:54:32.561522 17758 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:54:32.562240 17772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:32.562433 17758 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:54:32.562526 17758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/ts-2/wal
uuid: "d077c61ed5564832b616aa143e2f1894"
format_stamp: "Formatted at 2026-09-26 20:54:32 on dist-test-slave-hwlj"
server_key: "1cecd0b1d3697e2b43c44decebc0390c"
server_key_iv: "6d5af3aadfcd69d06bdd8d11e596ccde"
server_key_version: "encryptionkey@0"
I20260926 20:54:32.562601 17758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:54:32.579430 17758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:54:32.579820 17758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:54:32.579977 17758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:54:32.580231 17758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:54:32.580616 17758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:54:32.580665 17758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:32.580708 17758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:54:32.580734 17758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:32.586553 17758 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46571
I20260926 20:54:32.586596 17885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46571 every 8 connection(s)
I20260926 20:54:32.586947 17758 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
I20260926 20:54:32.590721 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 17758
I20260926 20:54:32.590835 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/ts-2/wal/instance
I20260926 20:54:32.591018 392 external_mini_cluster.cc:1428] Setting key 36c6fa9bf943540169ee67c6c1ea1326
I20260926 20:54:32.593291 17491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:32.587731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54587 (local address 127.0.98.62:40713)
0926 20:54:32.587818 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:32.587820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:32.588255 (+ 435us) server_negotiation.cc:407] Connection header received
0926 20:54:32.588920 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:32.588921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:32.588939 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:32.589022 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:32.589776 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:32.590289 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:32.591536 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:32.591835 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:32.591931 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:32.592982 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:32.592983 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:32.592983 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:54:32.592989 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:32.593005 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:54:32.593010 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:32.593065 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:32.593167 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 20:54:32.593203 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 20:54:32.593552 17888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:32.587825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40713 (local address 127.0.98.3:54587)
0926 20:54:32.588094 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:32.588134 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:32.588747 (+ 613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:32.589114 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:32.589118 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:32.589128 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:54:32.589693 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:32.589698 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:32.590385 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:32.590386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:32.591426 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:32.591430 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:32.591941 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:32.591942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:32.592092 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:32.592805 (+ 713us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:54:32.592806 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:54:32.592839 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:54:32.592864 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:54:32.593071 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:54:32.593076 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:54:32.593079 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:54:32.593172 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 20:54:32.593296 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":95,"threads_started":1}
I20260926 20:54:32.594049 17886 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40713
I20260926 20:54:32.594161 17886 heartbeater.cc:464] Registering TS with master...
I20260926 20:54:32.594365 17886 heartbeater.cc:511] Master 127.0.98.62:40713 requested a full tablet report, sending...
I20260926 20:54:32.594871 17436 ts_manager.cc:194] Registered new tserver with Master: d077c61ed5564832b616aa143e2f1894 (127.0.98.3:46571)
I20260926 20:54:32.595264 17436 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:54587
I20260926 20:54:32.602913 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:54:32.607383 17491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:32.604457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59128 (local address 127.0.98.62:40713)
0926 20:54:32.604530 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:32.604532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:32.604604 (+ 72us) server_negotiation.cc:407] Connection header received
0926 20:54:32.604685 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:32.604686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:32.604700 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:32.604750 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:32.604956 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:32.605459 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:32.606207 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:32.606421 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:32.606501 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:32.607119 (+ 618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:32.607121 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:32.607121 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:54:32.607127 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:32.607141 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:54:32.607145 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:32.607198 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:32.607257 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 20:54:32.607282 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 20:54:32.609095 17436 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59128:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:54:32.610692 17436 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:54:32.616822 17895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:32.612857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45635 (local address 127.0.0.1:59444)
0926 20:54:32.613147 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:32.613164 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:32.613232 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:32.613521 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:32.613521 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:32.613524 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:54:32.613574 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:32.613575 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:32.614881 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:32.614882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:32.615693 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:32.615695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:32.616512 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:32.616513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:32.616586 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:32.616598 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 20:54:32.616685 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 20:54:32.616737 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":1,"thread_start_us":100,"threads_started":1}
I20260926 20:54:32.616822 17898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:32.613095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59444 (local address 127.0.98.2:45635)
0926 20:54:32.613294 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:32.613299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:32.613309 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:54:32.613350 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:32.613352 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:32.613368 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:32.613454 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:54:32.613652 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:32.614807 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:32.615807 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:32.616418 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:32.616509 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:32.616599 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:32.616705 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 20:54:32.616740 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":87,"threads_started":1}
I20260926 20:54:32.617219 17896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:32.612857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46571 (local address 127.0.0.1:43740)
0926 20:54:32.613337 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:32.613346 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:32.613383 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:32.613573 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:32.613573 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:32.613575 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:54:32.613608 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:32.613610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:32.614855 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:32.614856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:32.615955 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:32.615957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:32.616951 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:32.616952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:32.617018 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:32.617022 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:54:32.617106 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 20:54:32.617142 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":26,"spinlock_wait_cycles":896,"thread_start_us":107,"threads_started":1}
I20260926 20:54:32.617475 17899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:32.613102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43740 (local address 127.0.98.3:46571)
0926 20:54:32.613383 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:32.613387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:32.613400 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:54:32.613436 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:32.613438 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:32.613451 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:32.613516 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:54:32.613681 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:32.614763 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:32.616036 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:32.616759 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:32.616821 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:32.616894 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:32.617337 (+ 443us) server_negotiation.cc:299] Negotiation successful
0926 20:54:32.617381 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":93,"threads_started":1}
I20260926 20:54:32.617580 17689 tablet_service.cc:1467] Processing CreateTablet for tablet e03f49e001d64beea312cf248bb7d593 (DEFAULT_TABLE table=test-table [id=c3135f5eb00d45a1a268a793f0165aa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:54:32.617779 17897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:32.612930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35666 (local address 127.0.98.1:38071)
0926 20:54:32.613330 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:32.613335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:32.613347 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:54:32.613390 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:32.613393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:32.613409 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:32.613484 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:54:32.613642 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:32.614471 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:32.615878 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:32.616624 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:32.616694 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:32.616781 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:32.617659 (+ 878us) server_negotiation.cc:299] Negotiation successful
0926 20:54:32.617692 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":115,"threads_started":1}
I20260926 20:54:32.617964 17894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:32.612847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38071 (local address 127.0.0.1:35666)
0926 20:54:32.613112 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:32.613177 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:32.613234 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:32.613484 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:32.613486 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:32.613488 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:54:32.613557 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:32.613559 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:32.614552 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:32.614553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:32.615779 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:32.615782 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:32.616829 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:32.616830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:32.617232 (+ 402us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:32.617236 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:54:32.617468 (+ 232us) client_negotiation.cc:240] Negotiation successful
0926 20:54:32.617901 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":60,"threads_started":1}
I20260926 20:54:32.617995 17689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e03f49e001d64beea312cf248bb7d593. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:54:32.618162 17820 tablet_service.cc:1467] Processing CreateTablet for tablet e03f49e001d64beea312cf248bb7d593 (DEFAULT_TABLE table=test-table [id=c3135f5eb00d45a1a268a793f0165aa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:54:32.618490 17820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e03f49e001d64beea312cf248bb7d593. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:54:32.618569 17558 tablet_service.cc:1467] Processing CreateTablet for tablet e03f49e001d64beea312cf248bb7d593 (DEFAULT_TABLE table=test-table [id=c3135f5eb00d45a1a268a793f0165aa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:54:32.618876 17558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e03f49e001d64beea312cf248bb7d593. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:54:32.620977 17900 tablet_bootstrap.cc:492] T e03f49e001d64beea312cf248bb7d593 P 8c869237bb14499982b2dd56e9b463c7: Bootstrap starting.
I20260926 20:54:32.621536 17900 tablet_bootstrap.cc:654] T e03f49e001d64beea312cf248bb7d593 P 8c869237bb14499982b2dd56e9b463c7: Neither blocks nor log segments found. Creating new log.
I20260926 20:54:32.621790 17901 tablet_bootstrap.cc:492] T e03f49e001d64beea312cf248bb7d593 P d077c61ed5564832b616aa143e2f1894: Bootstrap starting.
I20260926 20:54:32.621877 17900 log.cc:824] T e03f49e001d64beea312cf248bb7d593 P 8c869237bb14499982b2dd56e9b463c7: Log is configured to *not* fsync() on all Append() calls
I20260926 20:54:32.622170 17902 tablet_bootstrap.cc:492] T e03f49e001d64beea312cf248bb7d593 P 8c1618f4e070465aba788ee84db549ed: Bootstrap starting.
I20260926 20:54:32.622440 17901 tablet_bootstrap.cc:654] T e03f49e001d64beea312cf248bb7d593 P d077c61ed5564832b616aa143e2f1894: Neither blocks nor log segments found. Creating new log.
I20260926 20:54:32.622761 17901 log.cc:824] T e03f49e001d64beea312cf248bb7d593 P d077c61ed5564832b616aa143e2f1894: Log is configured to *not* fsync() on all Append() calls
I20260926 20:54:32.622808 17902 tablet_bootstrap.cc:654] T e03f49e001d64beea312cf248bb7d593 P 8c1618f4e070465aba788ee84db549ed: Neither blocks nor log segments found. Creating new log.
I20260926 20:54:32.623091 17902 log.cc:824] T e03f49e001d64beea312cf248bb7d593 P 8c1618f4e070465aba788ee84db549ed: Log is configured to *not* fsync() on all Append() calls
I20260926 20:54:32.623512 17901 tablet_bootstrap.cc:492] T e03f49e001d64beea312cf248bb7d593 P d077c61ed5564832b616aa143e2f1894: No bootstrap required, opened a new log
I20260926 20:54:32.623579 17901 ts_tablet_manager.cc:1397] T e03f49e001d64beea312cf248bb7d593 P d077c61ed5564832b616aa143e2f1894: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:54:32.623886 17900 tablet_bootstrap.cc:492] T e03f49e001d64beea312cf248bb7d593 P 8c869237bb14499982b2dd56e9b463c7: No bootstrap required, opened a new log
I20260926 20:54:32.623960 17900 ts_tablet_manager.cc:1397] T e03f49e001d64beea312cf248bb7d593 P 8c869237bb14499982b2dd56e9b463c7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 20:54:32.624171 17902 tablet_bootstrap.cc:492] T e03f49e001d64beea312cf248bb7d593 P 8c1618f4e070465aba788ee84db549ed: No bootstrap required, opened a new log
I20260926 20:54:32.624265 17902 ts_tablet_manager.cc:1397] T e03f49e001d64beea312cf248bb7d593 P 8c1618f4e070465aba788ee84db549ed: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:54:32.625530 17901 raft_consensus.cc:348] T e03f49e001d64beea312cf248bb7d593 P d077c61ed5564832b616aa143e2f1894 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1618f4e070465aba788ee84db549ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38071 } } peers { permanent_uuid: "8c869237bb14499982b2dd56e9b463c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45635 } } peers { permanent_uuid: "d077c61ed5564832b616aa143e2f1894" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46571 } }
I20260926 20:54:32.625661 17901 raft_consensus.cc:374] T e03f49e001d64beea312cf248bb7d593 P d077c61ed5564832b616aa143e2f1894 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:54:32.625684 17901 raft_consensus.cc:729] T e03f49e001d64beea312cf248bb7d593 P d077c61ed5564832b616aa143e2f1894 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d077c61ed5564832b616aa143e2f1894, State: Initialized, Role: FOLLOWER
I20260926 20:54:32.625835 17901 consensus_queue.cc:260] T e03f49e001d64beea312cf248bb7d593 P d077c61ed5564832b616aa143e2f1894 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1618f4e070465aba788ee84db549ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38071 } } peers { permanent_uuid: "8c869237bb14499982b2dd56e9b463c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45635 } } peers { permanent_uuid: "d077c61ed5564832b616aa143e2f1894" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46571 } }
I20260926 20:54:32.626093 17901 ts_tablet_manager.cc:1428] T e03f49e001d64beea312cf248bb7d593 P d077c61ed5564832b616aa143e2f1894: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 20:54:32.626107 17886 heartbeater.cc:503] Master 127.0.98.62:40713 was elected leader, sending a full tablet report...
I20260926 20:54:32.626663 17900 raft_consensus.cc:348] T e03f49e001d64beea312cf248bb7d593 P 8c869237bb14499982b2dd56e9b463c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1618f4e070465aba788ee84db549ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38071 } } peers { permanent_uuid: "8c869237bb14499982b2dd56e9b463c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45635 } } peers { permanent_uuid: "d077c61ed5564832b616aa143e2f1894" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46571 } }
I20260926 20:54:32.626835 17900 raft_consensus.cc:374] T e03f49e001d64beea312cf248bb7d593 P 8c869237bb14499982b2dd56e9b463c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:54:32.626870 17900 raft_consensus.cc:729] T e03f49e001d64beea312cf248bb7d593 P 8c869237bb14499982b2dd56e9b463c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c869237bb14499982b2dd56e9b463c7, State: Initialized, Role: FOLLOWER
I20260926 20:54:32.626987 17900 consensus_queue.cc:260] T e03f49e001d64beea312cf248bb7d593 P 8c869237bb14499982b2dd56e9b463c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1618f4e070465aba788ee84db549ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38071 } } peers { permanent_uuid: "8c869237bb14499982b2dd56e9b463c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45635 } } peers { permanent_uuid: "d077c61ed5564832b616aa143e2f1894" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46571 } }
I20260926 20:54:32.627063 17902 raft_consensus.cc:348] T e03f49e001d64beea312cf248bb7d593 P 8c1618f4e070465aba788ee84db549ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1618f4e070465aba788ee84db549ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38071 } } peers { permanent_uuid: "8c869237bb14499982b2dd56e9b463c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45635 } } peers { permanent_uuid: "d077c61ed5564832b616aa143e2f1894" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46571 } }
I20260926 20:54:32.627285 17755 heartbeater.cc:503] Master 127.0.98.62:40713 was elected leader, sending a full tablet report...
I20260926 20:54:32.627324 17902 raft_consensus.cc:374] T e03f49e001d64beea312cf248bb7d593 P 8c1618f4e070465aba788ee84db549ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:54:32.627351 17900 ts_tablet_manager.cc:1428] T e03f49e001d64beea312cf248bb7d593 P 8c869237bb14499982b2dd56e9b463c7: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 20:54:32.627384 17902 raft_consensus.cc:729] T e03f49e001d64beea312cf248bb7d593 P 8c1618f4e070465aba788ee84db549ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c1618f4e070465aba788ee84db549ed, State: Initialized, Role: FOLLOWER
I20260926 20:54:32.627521 17902 consensus_queue.cc:260] T e03f49e001d64beea312cf248bb7d593 P 8c1618f4e070465aba788ee84db549ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1618f4e070465aba788ee84db549ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38071 } } peers { permanent_uuid: "8c869237bb14499982b2dd56e9b463c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45635 } } peers { permanent_uuid: "d077c61ed5564832b616aa143e2f1894" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46571 } }
I20260926 20:54:32.627883 17624 heartbeater.cc:503] Master 127.0.98.62:40713 was elected leader, sending a full tablet report...
I20260926 20:54:32.627882 17902 ts_tablet_manager.cc:1428] T e03f49e001d64beea312cf248bb7d593 P 8c1618f4e070465aba788ee84db549ed: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260926 20:54:32.673141 17908 raft_consensus.cc:482] T e03f49e001d64beea312cf248bb7d593 P 8c1618f4e070465aba788ee84db549ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:54:32.673270 17908 raft_consensus.cc:504] T e03f49e001d64beea312cf248bb7d593 P 8c1618f4e070465aba788ee84db549ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1618f4e070465aba788ee84db549ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38071 } } peers { permanent_uuid: "8c869237bb14499982b2dd56e9b463c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45635 } } peers { permanent_uuid: "d077c61ed5564832b616aa143e2f1894" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46571 } }
I20260926 20:54:32.674039 17908 leader_election.cc:288] T e03f49e001d64beea312cf248bb7d593 P 8c1618f4e070465aba788ee84db549ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c869237bb14499982b2dd56e9b463c7 (127.0.98.2:45635), d077c61ed5564832b616aa143e2f1894 (127.0.98.3:46571)
I20260926 20:54:32.677547 17909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:32.674089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45635 (local address 127.0.98.1:37597)
0926 20:54:32.674270 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:32.674280 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:32.674368 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:32.674582 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:32.674583 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:32.674585 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:54:32.674639 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:32.674640 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:32.675626 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:32.675627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:32.676766 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:32.676768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:32.677273 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:32.677274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:32.677359 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:32.677363 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:54:32.677413 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 20:54:32.677443 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":10,"thread_start_us":118,"threads_started":1}
I20260926 20:54:32.677557 17898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:32.674150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37597 (local address 127.0.98.2:45635)
0926 20:54:32.674193 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:32.674195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:32.674340 (+ 145us) server_negotiation.cc:407] Connection header received
0926 20:54:32.674467 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:32.674469 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:32.674479 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:32.674536 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:54:32.674733 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:32.675549 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:32.676847 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:32.677194 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:32.677253 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:32.677261 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:32.677437 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 20:54:32.677489 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 20:54:32.677640 17626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:32.673992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46571 (local address 127.0.98.1:59393)
0926 20:54:32.674060 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:32.674074 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:32.674119 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:32.674399 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:32.674400 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:32.674402 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:54:32.674480 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:32.674482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:32.675502 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:32.675503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:32.676765 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:32.676768 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:32.677331 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:32.677333 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:32.677416 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:32.677420 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:54:32.677478 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 20:54:32.677517 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":1}
I20260926 20:54:32.677711 17899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:32.674039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59393 (local address 127.0.98.3:46571)
0926 20:54:32.674231 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:32.674233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:32.674244 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:54:32.674274 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:32.674275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:32.674286 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:32.674381 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:54:32.674568 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:32.675402 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:32.676878 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:32.677244 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:32.677309 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:32.677319 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:32.677500 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 20:54:32.677537 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169}
I20260926 20:54:32.677973 17709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e03f49e001d64beea312cf248bb7d593" candidate_uuid: "8c1618f4e070465aba788ee84db549ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c869237bb14499982b2dd56e9b463c7" is_pre_election: true
I20260926 20:54:32.678056 17840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e03f49e001d64beea312cf248bb7d593" candidate_uuid: "8c1618f4e070465aba788ee84db549ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d077c61ed5564832b616aa143e2f1894" is_pre_election: true
I20260926 20:54:32.678139 17709 raft_consensus.cc:2454] T e03f49e001d64beea312cf248bb7d593 P 8c869237bb14499982b2dd56e9b463c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c1618f4e070465aba788ee84db549ed in term 0.
I20260926 20:54:32.678238 17840 raft_consensus.cc:2454] T e03f49e001d64beea312cf248bb7d593 P d077c61ed5564832b616aa143e2f1894 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c1618f4e070465aba788ee84db549ed in term 0.
I20260926 20:54:32.678357 17513 leader_election.cc:302] T e03f49e001d64beea312cf248bb7d593 P 8c1618f4e070465aba788ee84db549ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c1618f4e070465aba788ee84db549ed, 8c869237bb14499982b2dd56e9b463c7; no voters:
I20260926 20:54:32.678512 17908 raft_consensus.cc:2781] T e03f49e001d64beea312cf248bb7d593 P 8c1618f4e070465aba788ee84db549ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:54:32.678556 17908 raft_consensus.cc:482] T e03f49e001d64beea312cf248bb7d593 P 8c1618f4e070465aba788ee84db549ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:54:32.678572 17908 raft_consensus.cc:3037] T e03f49e001d64beea312cf248bb7d593 P 8c1618f4e070465aba788ee84db549ed [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:54:32.679225 17908 raft_consensus.cc:504] T e03f49e001d64beea312cf248bb7d593 P 8c1618f4e070465aba788ee84db549ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1618f4e070465aba788ee84db549ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38071 } } peers { permanent_uuid: "8c869237bb14499982b2dd56e9b463c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45635 } } peers { permanent_uuid: "d077c61ed5564832b616aa143e2f1894" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46571 } }
I20260926 20:54:32.679369 17908 leader_election.cc:288] T e03f49e001d64beea312cf248bb7d593 P 8c1618f4e070465aba788ee84db549ed [CANDIDATE]: Term 1 election: Requested vote from peers 8c869237bb14499982b2dd56e9b463c7 (127.0.98.2:45635), d077c61ed5564832b616aa143e2f1894 (127.0.98.3:46571)
I20260926 20:54:32.679500 17709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e03f49e001d64beea312cf248bb7d593" candidate_uuid: "8c1618f4e070465aba788ee84db549ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c869237bb14499982b2dd56e9b463c7"
I20260926 20:54:32.679524 17840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e03f49e001d64beea312cf248bb7d593" candidate_uuid: "8c1618f4e070465aba788ee84db549ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d077c61ed5564832b616aa143e2f1894"
I20260926 20:54:32.679574 17709 raft_consensus.cc:3037] T e03f49e001d64beea312cf248bb7d593 P 8c869237bb14499982b2dd56e9b463c7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:54:32.679605 17840 raft_consensus.cc:3037] T e03f49e001d64beea312cf248bb7d593 P d077c61ed5564832b616aa143e2f1894 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:54:32.680285 17709 raft_consensus.cc:2454] T e03f49e001d64beea312cf248bb7d593 P 8c869237bb14499982b2dd56e9b463c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c1618f4e070465aba788ee84db549ed in term 1.
I20260926 20:54:32.680387 17840 raft_consensus.cc:2454] T e03f49e001d64beea312cf248bb7d593 P d077c61ed5564832b616aa143e2f1894 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c1618f4e070465aba788ee84db549ed in term 1.
I20260926 20:54:32.680438 17513 leader_election.cc:302] T e03f49e001d64beea312cf248bb7d593 P 8c1618f4e070465aba788ee84db549ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c1618f4e070465aba788ee84db549ed, 8c869237bb14499982b2dd56e9b463c7; no voters:
I20260926 20:54:32.680521 17908 raft_consensus.cc:2781] T e03f49e001d64beea312cf248bb7d593 P 8c1618f4e070465aba788ee84db549ed [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:54:32.680668 17908 raft_consensus.cc:686] T e03f49e001d64beea312cf248bb7d593 P 8c1618f4e070465aba788ee84db549ed [term 1 LEADER]: Becoming Leader. State: Replica: 8c1618f4e070465aba788ee84db549ed, State: Running, Role: LEADER
I20260926 20:54:32.680776 17908 consensus_queue.cc:237] T e03f49e001d64beea312cf248bb7d593 P 8c1618f4e070465aba788ee84db549ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1618f4e070465aba788ee84db549ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38071 } } peers { permanent_uuid: "8c869237bb14499982b2dd56e9b463c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45635 } } peers { permanent_uuid: "d077c61ed5564832b616aa143e2f1894" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46571 } }
I20260926 20:54:32.681617 17435 catalog_manager.cc:5495] T e03f49e001d64beea312cf248bb7d593 P 8c1618f4e070465aba788ee84db549ed reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c1618f4e070465aba788ee84db549ed (127.0.98.1). New cstate: current_term: 1 leader_uuid: "8c1618f4e070465aba788ee84db549ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1618f4e070465aba788ee84db549ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c869237bb14499982b2dd56e9b463c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d077c61ed5564832b616aa143e2f1894" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46571 } health_report { overall_health: UNKNOWN } } }
I20260926 20:54:32.693272 17897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:32.690926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35674 (local address 127.0.98.1:38071)
0926 20:54:32.690982 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:32.690984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:32.690993 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:54:32.691035 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:32.691036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:32.691052 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:32.691111 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:54:32.691225 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:32.691699 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:32.692518 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:32.692762 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:32.692823 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:32.692952 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:54:32.693050 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:54:32.693110 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:32.693140 (+ 30us) server_negotiation.cc:299] Negotiation successful
0926 20:54:32.693155 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 20:54:32.695106 17709 raft_consensus.cc:1261] T e03f49e001d64beea312cf248bb7d593 P 8c869237bb14499982b2dd56e9b463c7 [term 1 FOLLOWER]: Refusing update from remote peer 8c1618f4e070465aba788ee84db549ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:54:32.695452 17840 raft_consensus.cc:1261] T e03f49e001d64beea312cf248bb7d593 P d077c61ed5564832b616aa143e2f1894 [term 1 FOLLOWER]: Refusing update from remote peer 8c1618f4e070465aba788ee84db549ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:54:32.695477 17908 consensus_queue.cc:1035] T e03f49e001d64beea312cf248bb7d593 P 8c1618f4e070465aba788ee84db549ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c869237bb14499982b2dd56e9b463c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:54:32.695712 17908 consensus_queue.cc:1035] T e03f49e001d64beea312cf248bb7d593 P 8c1618f4e070465aba788ee84db549ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "d077c61ed5564832b616aa143e2f1894" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:54:32.697870 17913 mvcc.cc:205] Tried to move back new op lower bound from 7333708073756237824 to 7333708073701150720. Current Snapshot: MvccSnapshot[applied={T|T < 7333708073756237824}]
I20260926 20:54:32.700237 17434 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:54:32.700340 17434 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:54:32.701040 17434 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9dc7c9c1a36a496cb79887de8b4fb431: Sending DeleteTablet for 3 replicas of tablet e03f49e001d64beea312cf248bb7d593
I20260926 20:54:32.701377 17689 tablet_service.cc:1514] Processing DeleteTablet for tablet e03f49e001d64beea312cf248bb7d593 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:54:32 UTC) from {username='slave'} at 127.0.0.1:59444
I20260926 20:54:32.701400 17820 tablet_service.cc:1514] Processing DeleteTablet for tablet e03f49e001d64beea312cf248bb7d593 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:54:32 UTC) from {username='slave'} at 127.0.0.1:43740
I20260926 20:54:32.701633 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 17496
I20260926 20:54:32.701732 17558 tablet_service.cc:1514] Processing DeleteTablet for tablet e03f49e001d64beea312cf248bb7d593 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:54:32 UTC) from {username='slave'} at 127.0.0.1:35666
I20260926 20:54:32.702360 17924 tablet_replica.cc:330] T e03f49e001d64beea312cf248bb7d593 P d077c61ed5564832b616aa143e2f1894: stopping tablet replica
I20260926 20:54:32.702441 17924 raft_consensus.cc:2229] T e03f49e001d64beea312cf248bb7d593 P d077c61ed5564832b616aa143e2f1894 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:54:32.702553 17924 raft_consensus.cc:2258] T e03f49e001d64beea312cf248bb7d593 P d077c61ed5564832b616aa143e2f1894 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:54:32.702961 17924 ts_tablet_manager.cc:1905] T e03f49e001d64beea312cf248bb7d593 P d077c61ed5564832b616aa143e2f1894: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:54:32.703014 17923 tablet_replica.cc:330] T e03f49e001d64beea312cf248bb7d593 P 8c869237bb14499982b2dd56e9b463c7: stopping tablet replica
I20260926 20:54:32.703130 17923 raft_consensus.cc:2229] T e03f49e001d64beea312cf248bb7d593 P 8c869237bb14499982b2dd56e9b463c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:54:32.703240 17923 raft_consensus.cc:2258] T e03f49e001d64beea312cf248bb7d593 P 8c869237bb14499982b2dd56e9b463c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:54:32.703691 17923 ts_tablet_manager.cc:1905] T e03f49e001d64beea312cf248bb7d593 P 8c869237bb14499982b2dd56e9b463c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:54:32.704658 17924 ts_tablet_manager.cc:1918] T e03f49e001d64beea312cf248bb7d593 P d077c61ed5564832b616aa143e2f1894: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:54:32.704748 17924 log.cc:1196] T e03f49e001d64beea312cf248bb7d593 P d077c61ed5564832b616aa143e2f1894: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/ts-2/wal/wals/e03f49e001d64beea312cf248bb7d593
I20260926 20:54:32.705027 17924 ts_tablet_manager.cc:1939] T e03f49e001d64beea312cf248bb7d593 P d077c61ed5564832b616aa143e2f1894: Deleting consensus metadata
I20260926 20:54:32.705477 17420 catalog_manager.cc:4841] TS d077c61ed5564832b616aa143e2f1894 (127.0.98.3:46571): tablet e03f49e001d64beea312cf248bb7d593 (table test-table [id=c3135f5eb00d45a1a268a793f0165aa4]) successfully deleted
I20260926 20:54:32.705729 17923 ts_tablet_manager.cc:1918] T e03f49e001d64beea312cf248bb7d593 P 8c869237bb14499982b2dd56e9b463c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:54:32.705811 17923 log.cc:1196] T e03f49e001d64beea312cf248bb7d593 P 8c869237bb14499982b2dd56e9b463c7: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455725523895-392-0/minicluster-data/ts-1/wal/wals/e03f49e001d64beea312cf248bb7d593
I20260926 20:54:32.706058 17923 ts_tablet_manager.cc:1939] T e03f49e001d64beea312cf248bb7d593 P 8c869237bb14499982b2dd56e9b463c7: Deleting consensus metadata
I20260926 20:54:32.706830 17422 catalog_manager.cc:4841] TS 8c869237bb14499982b2dd56e9b463c7 (127.0.98.2:45635): tablet e03f49e001d64beea312cf248bb7d593 (table test-table [id=c3135f5eb00d45a1a268a793f0165aa4]) successfully deleted
W20260926 20:54:32.708798 17423 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38071 (error 108)
I20260926 20:54:32.708927 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 17627
I20260926 20:54:32.709715 17894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:32.709569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38071 (local address 127.0.0.1:35680)
0926 20:54:32.709627 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:32.709673 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 20:54:32.709853 17423 catalog_manager.cc:4584] TS 8c1618f4e070465aba788ee84db549ed (127.0.98.1:38071): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e03f49e001d64beea312cf248bb7d593: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38071: connect: Connection refused (error 111)
I20260926 20:54:32.713961 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 17758
I20260926 20:54:32.718691 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 17405
2026-09-26T20:54:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (681 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:54:32 dist-test-slave-hwlj krb5kdc[17928](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:54:32 dist-test-slave-hwlj krb5kdc[17928](info): set up 2 sockets
Sep 26 20:54:32 dist-test-slave-hwlj krb5kdc[17928](info): commencing operation
krb5kdc: starting...
W20260926 20:54:33.755409 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/krb5kdc/test-user.keytab.
Sep 26 20:54:33 dist-test-slave-hwlj krb5kdc[17928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:54:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:54:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 20:54:33.875017 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34565
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34565
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:54:33.956749 17941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:54:33.956964 17941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:54:33.956986 17941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:54:33.958173 17941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:54:33.958215 17941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:54:33.958230 17941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:54:33.958242 17941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:54:33.958254 17941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:54:33.959740 17941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:54:33.960016 17941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:54:33.960242 17941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:54:33.965140 17949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:33.965140 17946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:33.965322 17941 server_base.cc:1034] running on GCE node
W20260926 20:54:33.965163 17947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:33.965651 17941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:54:33.965982 17941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:54:33.967144 17941 hybrid_clock.cc:648] HybridClock initialized: now 1790456073967119 us; error 39 us; skew 500 ppm
Sep 26 20:54:33 dist-test-slave-hwlj krb5kdc[17928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:54:33.969959 17941 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:54:33.970489 17941 webserver.cc:460] Webserver started at http://127.0.98.62:35517/ using document root <none> and password file <none>
I20260926 20:54:33.970696 17941 fs_manager.cc:359] Metadata directory not provided
I20260926 20:54:33.970741 17941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:54:33.970842 17941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:54:33.972043 17941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/master-0/data/instance:
uuid: "d5ac981d319d4420b09bebdb8c1a54aa"
format_stamp: "Formatted at 2026-09-26 20:54:33 on dist-test-slave-hwlj"
server_key: "90455328c149c1f9b7e4f44313e3d305"
server_key_iv: "83ea4971f1ef946c24089fb101235c0f"
server_key_version: "encryptionkey@0"
I20260926 20:54:33.972381 17941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/master-0/wal/instance:
uuid: "d5ac981d319d4420b09bebdb8c1a54aa"
format_stamp: "Formatted at 2026-09-26 20:54:33 on dist-test-slave-hwlj"
server_key: "90455328c149c1f9b7e4f44313e3d305"
server_key_iv: "83ea4971f1ef946c24089fb101235c0f"
server_key_version: "encryptionkey@0"
I20260926 20:54:33.973798 17941 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:54:33.974570 17956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:33.974751 17941 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:54:33.974843 17941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "d5ac981d319d4420b09bebdb8c1a54aa"
format_stamp: "Formatted at 2026-09-26 20:54:33 on dist-test-slave-hwlj"
server_key: "90455328c149c1f9b7e4f44313e3d305"
server_key_iv: "83ea4971f1ef946c24089fb101235c0f"
server_key_version: "encryptionkey@0"
I20260926 20:54:33.974902 17941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:54:33.991164 17941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:54:33.993204 17941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:54:33.993311 17941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:54:33.997239 17941 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34565
I20260926 20:54:33.997311 18008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34565 every 8 connection(s)
I20260926 20:54:33.997642 17941 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:54:33.998538 18009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:54:33.999892 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 17941
I20260926 20:54:34.000015 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/master-0/wal/instance
I20260926 20:54:34.000216 392 external_mini_cluster.cc:1428] Setting key ba6f7902eb63ebd39dcede6939c9f92f
I20260926 20:54:34.001362 18009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5ac981d319d4420b09bebdb8c1a54aa: Bootstrap starting.
I20260926 20:54:34.002108 18009 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5ac981d319d4420b09bebdb8c1a54aa: Neither blocks nor log segments found. Creating new log.
I20260926 20:54:34.002465 18009 log.cc:824] T 00000000000000000000000000000000 P d5ac981d319d4420b09bebdb8c1a54aa: Log is configured to *not* fsync() on all Append() calls
I20260926 20:54:34.003168 18009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5ac981d319d4420b09bebdb8c1a54aa: No bootstrap required, opened a new log
I20260926 20:54:34.005627 18009 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5ac981d319d4420b09bebdb8c1a54aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ac981d319d4420b09bebdb8c1a54aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34565 } }
I20260926 20:54:34.005836 18009 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5ac981d319d4420b09bebdb8c1a54aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:54:34.005883 18009 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5ac981d319d4420b09bebdb8c1a54aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5ac981d319d4420b09bebdb8c1a54aa, State: Initialized, Role: FOLLOWER
I20260926 20:54:34.006024 18009 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5ac981d319d4420b09bebdb8c1a54aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ac981d319d4420b09bebdb8c1a54aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34565 } }
I20260926 20:54:34.006112 18009 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5ac981d319d4420b09bebdb8c1a54aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:54:34.006147 18009 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5ac981d319d4420b09bebdb8c1a54aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:54:34.006202 18009 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5ac981d319d4420b09bebdb8c1a54aa [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:54:34.006996 18009 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5ac981d319d4420b09bebdb8c1a54aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ac981d319d4420b09bebdb8c1a54aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34565 } }
I20260926 20:54:34.007112 18009 leader_election.cc:302] T 00000000000000000000000000000000 P d5ac981d319d4420b09bebdb8c1a54aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5ac981d319d4420b09bebdb8c1a54aa; no voters:
I20260926 20:54:34.007319 18009 leader_election.cc:288] T 00000000000000000000000000000000 P d5ac981d319d4420b09bebdb8c1a54aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:54:34.007344 18014 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5ac981d319d4420b09bebdb8c1a54aa [term 1 FOLLOWER]: Leader election won for term 1
Sep 26 20:54:34 dist-test-slave-hwlj krb5kdc[17928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:54:34.007474 18014 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5ac981d319d4420b09bebdb8c1a54aa [term 1 LEADER]: Becoming Leader. State: Replica: d5ac981d319d4420b09bebdb8c1a54aa, State: Running, Role: LEADER
I20260926 20:54:34.007598 18009 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5ac981d319d4420b09bebdb8c1a54aa [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:54:34.008097 18014 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5ac981d319d4420b09bebdb8c1a54aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ac981d319d4420b09bebdb8c1a54aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34565 } }
I20260926 20:54:34.008518 18016 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5ac981d319d4420b09bebdb8c1a54aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5ac981d319d4420b09bebdb8c1a54aa. Latest consensus state: current_term: 1 leader_uuid: "d5ac981d319d4420b09bebdb8c1a54aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ac981d319d4420b09bebdb8c1a54aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34565 } } }
I20260926 20:54:34.008677 18015 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5ac981d319d4420b09bebdb8c1a54aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5ac981d319d4420b09bebdb8c1a54aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ac981d319d4420b09bebdb8c1a54aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34565 } } }
I20260926 20:54:34.008780 18015 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5ac981d319d4420b09bebdb8c1a54aa [sys.catalog]: This master's current role is: LEADER
I20260926 20:54:34.009138 18028 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:54:34.008677 18016 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5ac981d319d4420b09bebdb8c1a54aa [sys.catalog]: This master's current role is: LEADER
I20260926 20:54:34.009617 18028 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:54:34.011294 18028 catalog_manager.cc:1347] Generated new cluster ID: 8feff78b947e4a79a7c38f8782c8ab04
I20260926 20:54:34.011349 18028 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:54:34.012115 18012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:34.000740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42934 (local address 127.0.98.62:34565)
0926 20:54:34.000930 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:34.000939 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:34.000972 (+ 33us) server_negotiation.cc:407] Connection header received
0926 20:54:34.001363 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:34.001369 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:34.001437 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:34.001572 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:34.002430 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:34.003525 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:34.004638 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:34.005037 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:34.005114 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:34.008046 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:34.008049 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:34.008050 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:54:34.008061 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:34.010308 (+ 2247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:34.010715 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:34.010718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:34.010718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:34.010819 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:34.011075 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:34.011076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:34.011077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:34.011430 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:34.011589 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:34.011728 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 20:54:34.011892 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":105,"threads_started":1}
I20260926 20:54:34.030689 18028 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:54:34.031248 18028 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:54:34.035940 18028 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5ac981d319d4420b09bebdb8c1a54aa: Generated new TSK 0
I20260926 20:54:34.036130 18028 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 20:54:34.067008 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34565
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:54:34.151083 18035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:54:34.151278 18035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:54:34.151301 18035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:54:34.152388 18035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:54:34.152424 18035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:54:34.152450 18035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:54:34.153854 18035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:54:34.154034 18035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:54:34.154211 18035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:54:34.159317 18041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:34.159451 18035 server_base.cc:1034] running on GCE node
W20260926 20:54:34.159403 18040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:34.159313 18043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:34.159823 18035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:54:34.160081 18035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:54:34.161250 18035 hybrid_clock.cc:648] HybridClock initialized: now 1790456074161246 us; error 47 us; skew 500 ppm
Sep 26 20:54:34 dist-test-slave-hwlj krb5kdc[17928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456074, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:54:34.163740 18035 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 20:54:34.164283 18035 webserver.cc:460] Webserver started at http://127.0.98.1:40749/ using document root <none> and password file <none>
I20260926 20:54:34.164505 18035 fs_manager.cc:359] Metadata directory not provided
I20260926 20:54:34.164568 18035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:54:34.164711 18035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:54:34.165727 18035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/ts-0/data/instance:
uuid: "05cf556da4b74facb5d2aa28b75d4dd8"
format_stamp: "Formatted at 2026-09-26 20:54:34 on dist-test-slave-hwlj"
server_key: "a3aa8723241cc1d99190f1a873584f2b"
server_key_iv: "bcf6d84a58b0886629f6b29f00b0bf0f"
server_key_version: "encryptionkey@0"
I20260926 20:54:34.166085 18035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/ts-0/wal/instance:
uuid: "05cf556da4b74facb5d2aa28b75d4dd8"
format_stamp: "Formatted at 2026-09-26 20:54:34 on dist-test-slave-hwlj"
server_key: "a3aa8723241cc1d99190f1a873584f2b"
server_key_iv: "bcf6d84a58b0886629f6b29f00b0bf0f"
server_key_version: "encryptionkey@0"
I20260926 20:54:34.167364 18035 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 20:54:34.168134 18050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:34.168301 18035 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:34.168394 18035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/ts-0/wal
uuid: "05cf556da4b74facb5d2aa28b75d4dd8"
format_stamp: "Formatted at 2026-09-26 20:54:34 on dist-test-slave-hwlj"
server_key: "a3aa8723241cc1d99190f1a873584f2b"
server_key_iv: "bcf6d84a58b0886629f6b29f00b0bf0f"
server_key_version: "encryptionkey@0"
I20260926 20:54:34.168473 18035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:54:34.188472 18035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:54:34.190492 18035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:54:34.190663 18035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:54:34.190932 18035 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:54:34.191250 18035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:54:34.191304 18035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:34.191344 18035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:54:34.191385 18035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:34.197367 18035 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35347
I20260926 20:54:34.197415 18163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35347 every 8 connection(s)
I20260926 20:54:34.197739 18035 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:54:34.202327 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 18035
I20260926 20:54:34.202425 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/ts-0/wal/instance
I20260926 20:54:34.202661 392 external_mini_cluster.cc:1428] Setting key 8980ad090e36ebf3bbbadb8259726501
Sep 26 20:54:34 dist-test-slave-hwlj krb5kdc[17928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456074, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:54:34.207580 18012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:34.198414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47043 (local address 127.0.98.62:34565)
0926 20:54:34.198478 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:34.198480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:34.198714 (+ 234us) server_negotiation.cc:407] Connection header received
0926 20:54:34.199683 (+ 969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:34.199685 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:34.199697 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:34.199781 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:34.200423 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:34.200950 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:34.201969 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:34.202198 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:34.202291 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:34.204653 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:34.204654 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:34.204655 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:54:34.204663 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:34.206072 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:34.206497 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:34.206499 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:34.206499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:34.206562 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:34.206832 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:34.206833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:34.206833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:34.206952 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:34.207027 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:34.207336 (+ 309us) server_negotiation.cc:299] Negotiation successful
0926 20:54:34.207435 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 20:54:34.207818 18166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:34.198419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34565 (local address 127.0.98.1:47043)
0926 20:54:34.198585 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:34.198623 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:34.199569 (+ 946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:34.199864 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:34.199867 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:34.200129 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:54:34.200326 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:34.200329 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:34.201047 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:34.201048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:34.201875 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:34.201877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:34.202295 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:34.202296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:34.202468 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:34.203086 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:54:34.203095 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:54:34.204544 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:54:34.206173 (+ 1629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:54:34.206182 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:54:34.206185 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:54:34.206423 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:54:34.206626 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:54:34.206627 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:54:34.206628 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:54:34.206746 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:54:34.207018 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:54:34.207025 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:54:34.207199 (+ 174us) client_negotiation.cc:769] Sending connection context
0926 20:54:34.207312 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 20:54:34.207486 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":82,"threads_started":1}
I20260926 20:54:34.208401 18164 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34565
I20260926 20:54:34.208520 18164 heartbeater.cc:464] Registering TS with master...
I20260926 20:54:34.208753 18164 heartbeater.cc:511] Master 127.0.98.62:34565 requested a full tablet report, sending...
I20260926 20:54:34.209372 17973 ts_manager.cc:194] Registered new tserver with Master: 05cf556da4b74facb5d2aa28b75d4dd8 (127.0.98.1:35347)
I20260926 20:54:34.210345 17973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47043
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 20:54:34.230690 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34565
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:54:34.311658 18169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:54:34.311930 18169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:54:34.311955 18169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:54:34.313040 18169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:54:34.313076 18169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:54:34.313100 18169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:54:34.314524 18169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:54:34.314728 18169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:54:34.314962 18169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:54:34.319829 18177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:34.319869 18175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:34.319898 18169 server_base.cc:1034] running on GCE node
W20260926 20:54:34.319895 18174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:34.320305 18169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:54:34.320552 18169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:54:34.321715 18169 hybrid_clock.cc:648] HybridClock initialized: now 1790456074321705 us; error 28 us; skew 500 ppm
Sep 26 20:54:34 dist-test-slave-hwlj krb5kdc[17928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456074, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:54:34.324316 18169 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 20:54:34.324705 18169 webserver.cc:460] Webserver started at http://127.0.98.2:45195/ using document root <none> and password file <none>
I20260926 20:54:34.324901 18169 fs_manager.cc:359] Metadata directory not provided
I20260926 20:54:34.324941 18169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:54:34.325073 18169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:54:34.326059 18169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/ts-1/data/instance:
uuid: "30cf8f26f73c4217a6a52f8586265ef2"
format_stamp: "Formatted at 2026-09-26 20:54:34 on dist-test-slave-hwlj"
server_key: "d775438143923336fc55ac000d97972d"
server_key_iv: "462bf88e8f2ff1a616d687f617f98760"
server_key_version: "encryptionkey@0"
I20260926 20:54:34.326360 18169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/ts-1/wal/instance:
uuid: "30cf8f26f73c4217a6a52f8586265ef2"
format_stamp: "Formatted at 2026-09-26 20:54:34 on dist-test-slave-hwlj"
server_key: "d775438143923336fc55ac000d97972d"
server_key_iv: "462bf88e8f2ff1a616d687f617f98760"
server_key_version: "encryptionkey@0"
I20260926 20:54:34.327562 18169 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:54:34.328312 18184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:34.328493 18169 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:34.328562 18169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/ts-1/wal
uuid: "30cf8f26f73c4217a6a52f8586265ef2"
format_stamp: "Formatted at 2026-09-26 20:54:34 on dist-test-slave-hwlj"
server_key: "d775438143923336fc55ac000d97972d"
server_key_iv: "462bf88e8f2ff1a616d687f617f98760"
server_key_version: "encryptionkey@0"
I20260926 20:54:34.328615 18169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:54:34.341336 18169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:54:34.343616 18169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:54:34.343729 18169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:54:34.343973 18169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:54:34.344276 18169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:54:34.344311 18169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:34.344362 18169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:54:34.344393 18169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:34.350207 18169 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41937
I20260926 20:54:34.350247 18297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41937 every 8 connection(s)
I20260926 20:54:34.350564 18169 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
I20260926 20:54:34.355589 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 18169
I20260926 20:54:34.356060 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/ts-1/wal/instance
I20260926 20:54:34.356319 392 external_mini_cluster.cc:1428] Setting key fd5f69ab69b8191cd67f862a27bdbd07
Sep 26 20:54:34 dist-test-slave-hwlj krb5kdc[17928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456074, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:54:34.361088 18012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:34.351305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40237 (local address 127.0.98.62:34565)
0926 20:54:34.351357 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:34.351360 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:34.351580 (+ 220us) server_negotiation.cc:407] Connection header received
0926 20:54:34.352766 (+ 1186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:34.352768 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:34.352782 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:34.352829 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:34.353481 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:34.353991 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:34.355015 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:34.355227 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:34.355309 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:34.358044 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:34.358045 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:34.358046 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:54:34.358052 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:34.359705 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:34.360155 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:34.360156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:34.360156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:34.360215 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:34.360455 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:34.360456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:34.360456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:34.360577 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:34.360649 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:34.360912 (+ 263us) server_negotiation.cc:299] Negotiation successful
0926 20:54:34.360993 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 20:54:34.361284 18300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:34.351233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34565 (local address 127.0.98.2:40237)
0926 20:54:34.351442 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:34.351494 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:34.352638 (+ 1144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:34.352915 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:34.352918 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:34.353181 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:54:34.353385 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:34.353389 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:34.354092 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:34.354093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:34.354923 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:34.354925 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:34.355319 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:34.355320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:34.355492 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:34.356116 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:54:34.356127 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:54:34.357961 (+ 1834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:54:34.359811 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:54:34.359840 (+ 29us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:54:34.359843 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:54:34.360062 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:54:34.360282 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:54:34.360283 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:54:34.360284 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:54:34.360389 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:54:34.360639 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:54:34.360646 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:54:34.360781 (+ 135us) client_negotiation.cc:769] Sending connection context
0926 20:54:34.360893 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 20:54:34.361058 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":83,"threads_started":1}
I20260926 20:54:34.361809 18298 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34565
I20260926 20:54:34.361925 18298 heartbeater.cc:464] Registering TS with master...
I20260926 20:54:34.362155 18298 heartbeater.cc:511] Master 127.0.98.62:34565 requested a full tablet report, sending...
I20260926 20:54:34.362540 17973 ts_manager.cc:194] Registered new tserver with Master: 30cf8f26f73c4217a6a52f8586265ef2 (127.0.98.2:41937)
I20260926 20:54:34.362931 17973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40237
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 20:54:34.383605 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34565
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:54:34.464641 18303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:54:34.464862 18303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:54:34.464885 18303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:54:34.465988 18303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:54:34.466027 18303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:54:34.466053 18303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:54:34.467463 18303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:54:34.467646 18303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:54:34.467908 18303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:54:34.473183 18311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:34.473189 18309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:34.473344 18308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:34.473534 18303 server_base.cc:1034] running on GCE node
I20260926 20:54:34.473773 18303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:54:34.474020 18303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:54:34.475196 18303 hybrid_clock.cc:648] HybridClock initialized: now 1790456074475172 us; error 52 us; skew 500 ppm
Sep 26 20:54:34 dist-test-slave-hwlj krb5kdc[17928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456074, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:54:34.477885 18303 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 20:54:34.478436 18303 webserver.cc:460] Webserver started at http://127.0.98.3:37723/ using document root <none> and password file <none>
I20260926 20:54:34.478664 18303 fs_manager.cc:359] Metadata directory not provided
I20260926 20:54:34.478737 18303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:54:34.478940 18303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:54:34.480299 18303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/ts-2/data/instance:
uuid: "d910ebb4b7b04c56b7b4c2643f0ae48a"
format_stamp: "Formatted at 2026-09-26 20:54:34 on dist-test-slave-hwlj"
server_key: "4d39b51eea2ae0e0f9ea93654b320fe6"
server_key_iv: "040f6b192b2fc21523bfccd8e2c33fb3"
server_key_version: "encryptionkey@0"
I20260926 20:54:34.480686 18303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d910ebb4b7b04c56b7b4c2643f0ae48a"
format_stamp: "Formatted at 2026-09-26 20:54:34 on dist-test-slave-hwlj"
server_key: "4d39b51eea2ae0e0f9ea93654b320fe6"
server_key_iv: "040f6b192b2fc21523bfccd8e2c33fb3"
server_key_version: "encryptionkey@0"
I20260926 20:54:34.482122 18303 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:54:34.482910 18318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:34.483072 18303 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:54:34.483191 18303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/ts-2/wal
uuid: "d910ebb4b7b04c56b7b4c2643f0ae48a"
format_stamp: "Formatted at 2026-09-26 20:54:34 on dist-test-slave-hwlj"
server_key: "4d39b51eea2ae0e0f9ea93654b320fe6"
server_key_iv: "040f6b192b2fc21523bfccd8e2c33fb3"
server_key_version: "encryptionkey@0"
I20260926 20:54:34.483270 18303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:54:34.492111 18303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:54:34.494115 18303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:54:34.494302 18303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:54:34.494534 18303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:54:34.494861 18303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:54:34.494921 18303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:34.494967 18303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:54:34.494997 18303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:34.501358 18303 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34421
I20260926 20:54:34.501417 18431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34421 every 8 connection(s)
I20260926 20:54:34.501757 18303 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 20:54:34 dist-test-slave-hwlj krb5kdc[17928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456074, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:54:34.508538 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 18303
I20260926 20:54:34.508687 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/ts-2/wal/instance
I20260926 20:54:34.508905 392 external_mini_cluster.cc:1428] Setting key 67139f34c000cacad3c0b94f611825cc
I20260926 20:54:34.511564 18012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:34.502468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33307 (local address 127.0.98.62:34565)
0926 20:54:34.502530 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:34.502532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:34.502704 (+ 172us) server_negotiation.cc:407] Connection header received
0926 20:54:34.503702 (+ 998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:34.503704 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:34.503718 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:34.503842 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:34.504420 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:34.504997 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:34.505963 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:34.506153 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:34.506215 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:34.508605 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:34.508607 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:34.508608 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:54:34.508615 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:34.510232 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:34.510640 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:34.510642 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:34.510643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:34.510702 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:34.510940 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:34.510941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:34.510941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:34.511057 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:34.511128 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:34.511399 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 20:54:34.511475 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 20:54:34.511744 18434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:34.502448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34565 (local address 127.0.98.3:33307)
0926 20:54:34.502602 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:34.502634 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:34.503598 (+ 964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:34.503896 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:34.503899 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:34.504152 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:54:34.504337 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:34.504341 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:34.505077 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:34.505078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:34.505874 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:34.505876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:34.506226 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:34.506227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:34.506323 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:34.506843 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:54:34.506851 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:54:34.508516 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:54:34.510318 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:54:34.510324 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:54:34.510326 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:54:34.510543 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:54:34.510764 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:54:34.510765 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:54:34.510766 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:54:34.510875 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:54:34.511120 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:54:34.511125 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:54:34.511262 (+ 137us) client_negotiation.cc:769] Sending connection context
0926 20:54:34.511365 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 20:54:34.511520 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":92,"threads_started":1}
I20260926 20:54:34.512238 18432 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34565
I20260926 20:54:34.512331 18432 heartbeater.cc:464] Registering TS with master...
I20260926 20:54:34.512470 18432 heartbeater.cc:511] Master 127.0.98.62:34565 requested a full tablet report, sending...
I20260926 20:54:34.512755 17973 ts_manager.cc:194] Registered new tserver with Master: d910ebb4b7b04c56b7b4c2643f0ae48a (127.0.98.3:34421)
I20260926 20:54:34.513074 17973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33307
I20260926 20:54:34.520604 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:54:34.527835 18012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:34.521911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42946 (local address 127.0.98.62:34565)
0926 20:54:34.521967 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:34.521969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:34.522069 (+ 100us) server_negotiation.cc:407] Connection header received
0926 20:54:34.522192 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:34.522194 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:34.522207 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:34.522264 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:34.522820 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:34.523270 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:34.524024 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:34.524200 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:34.524246 (+ 46us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:34.525470 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:34.525472 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:34.525472 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:54:34.525478 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:34.526802 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:34.527143 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:34.527144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:34.527144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:34.527192 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:34.527364 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:34.527365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:34.527365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:34.527481 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:34.527556 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:34.527649 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 20:54:34.527696 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 20:54:34.529362 17973 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42946:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:54:34.530866 17973 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:54:34.536716 18443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:34.533118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55650 (local address 127.0.98.2:41937)
0926 20:54:34.533258 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:34.533263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:34.533274 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:54:34.533315 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:34.533317 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:34.533333 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:34.533422 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:54:34.533583 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:34.534410 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:34.535543 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:34.536315 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:34.536391 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:34.536453 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:34.536600 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 20:54:34.536638 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":74,"threads_started":1}
I20260926 20:54:34.536718 18441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:34.532936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41937 (local address 127.0.0.1:55650)
0926 20:54:34.533087 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:34.533139 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:34.533204 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:34.533427 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:34.533429 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:34.533431 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:54:34.533487 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:34.533488 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:34.534508 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:34.534509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:34.535457 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:34.535459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:34.536402 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:34.536403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:34.536485 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:34.536500 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 20:54:34.536578 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 20:54:34.536634 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":55,"threads_started":1}
I20260926 20:54:34.537143 18440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:34.532936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35347 (local address 127.0.0.1:34004)
0926 20:54:34.533298 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:34.533307 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:34.533342 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:34.533841 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:34.533843 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:34.533844 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:54:34.533884 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:34.533886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:34.534697 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:34.534698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:34.536008 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:34.536010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:34.536903 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:34.536904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:34.536971 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:34.536976 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:54:34.537040 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 20:54:34.537080 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":2,"thread_start_us":51,"threads_started":1}
I20260926 20:54:34.537168 18445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:34.533297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34004 (local address 127.0.98.1:35347)
0926 20:54:34.533679 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:34.533683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:34.533695 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:54:34.533732 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:34.533734 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:34.533749 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:34.533833 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:54:34.533945 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:34.534593 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:34.536096 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:34.536730 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:34.536811 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:34.536869 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:34.537056 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 20:54:34.537096 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":88,"threads_started":1}
I20260926 20:54:34.537566 18232 tablet_service.cc:1467] Processing CreateTablet for tablet 0f79ab7291dd4c1ab96dfc80a1579408 (DEFAULT_TABLE table=test-table [id=adcdb66c4e3b4937a84cd17d2807d072]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:54:34.537807 18098 tablet_service.cc:1467] Processing CreateTablet for tablet 0f79ab7291dd4c1ab96dfc80a1579408 (DEFAULT_TABLE table=test-table [id=adcdb66c4e3b4937a84cd17d2807d072]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:54:34.537966 18232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f79ab7291dd4c1ab96dfc80a1579408. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:54:34.538132 18098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f79ab7291dd4c1ab96dfc80a1579408. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:54:34.538710 18444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:34.533188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40490 (local address 127.0.98.3:34421)
0926 20:54:34.534543 (+ 1355us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:34.534547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:34.534556 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:54:34.534594 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:34.534597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:34.534612 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:34.534681 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:54:34.534884 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:34.535945 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:34.537619 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:34.538352 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:34.538428 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:34.538482 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:34.538588 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 20:54:34.538629 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":85,"threads_started":1}
I20260926 20:54:34.538676 18442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:34.532941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34421 (local address 127.0.0.1:40490)
0926 20:54:34.533468 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:34.533479 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:34.533526 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:34.534760 (+ 1234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:34.534761 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:34.534763 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:54:34.534797 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:34.534798 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:34.536050 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:34.536051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:34.537520 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:34.537523 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:34.538439 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:34.538440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:34.538506 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:34.538510 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:54:34.538564 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 20:54:34.538606 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":75,"threads_started":1}
I20260926 20:54:34.539453 18366 tablet_service.cc:1467] Processing CreateTablet for tablet 0f79ab7291dd4c1ab96dfc80a1579408 (DEFAULT_TABLE table=test-table [id=adcdb66c4e3b4937a84cd17d2807d072]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:54:34.539824 18366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f79ab7291dd4c1ab96dfc80a1579408. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:54:34.541106 18446 tablet_bootstrap.cc:492] T 0f79ab7291dd4c1ab96dfc80a1579408 P 30cf8f26f73c4217a6a52f8586265ef2: Bootstrap starting.
I20260926 20:54:34.541473 18447 tablet_bootstrap.cc:492] T 0f79ab7291dd4c1ab96dfc80a1579408 P 05cf556da4b74facb5d2aa28b75d4dd8: Bootstrap starting.
I20260926 20:54:34.541843 18446 tablet_bootstrap.cc:654] T 0f79ab7291dd4c1ab96dfc80a1579408 P 30cf8f26f73c4217a6a52f8586265ef2: Neither blocks nor log segments found. Creating new log.
I20260926 20:54:34.542128 18447 tablet_bootstrap.cc:654] T 0f79ab7291dd4c1ab96dfc80a1579408 P 05cf556da4b74facb5d2aa28b75d4dd8: Neither blocks nor log segments found. Creating new log.
I20260926 20:54:34.542440 18446 log.cc:824] T 0f79ab7291dd4c1ab96dfc80a1579408 P 30cf8f26f73c4217a6a52f8586265ef2: Log is configured to *not* fsync() on all Append() calls
I20260926 20:54:34.542716 18447 log.cc:824] T 0f79ab7291dd4c1ab96dfc80a1579408 P 05cf556da4b74facb5d2aa28b75d4dd8: Log is configured to *not* fsync() on all Append() calls
I20260926 20:54:34.542829 18449 tablet_bootstrap.cc:492] T 0f79ab7291dd4c1ab96dfc80a1579408 P d910ebb4b7b04c56b7b4c2643f0ae48a: Bootstrap starting.
I20260926 20:54:34.543309 18446 tablet_bootstrap.cc:492] T 0f79ab7291dd4c1ab96dfc80a1579408 P 30cf8f26f73c4217a6a52f8586265ef2: No bootstrap required, opened a new log
I20260926 20:54:34.543393 18446 ts_tablet_manager.cc:1397] T 0f79ab7291dd4c1ab96dfc80a1579408 P 30cf8f26f73c4217a6a52f8586265ef2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:54:34.543434 18447 tablet_bootstrap.cc:492] T 0f79ab7291dd4c1ab96dfc80a1579408 P 05cf556da4b74facb5d2aa28b75d4dd8: No bootstrap required, opened a new log
I20260926 20:54:34.543465 18449 tablet_bootstrap.cc:654] T 0f79ab7291dd4c1ab96dfc80a1579408 P d910ebb4b7b04c56b7b4c2643f0ae48a: Neither blocks nor log segments found. Creating new log.
I20260926 20:54:34.543506 18447 ts_tablet_manager.cc:1397] T 0f79ab7291dd4c1ab96dfc80a1579408 P 05cf556da4b74facb5d2aa28b75d4dd8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:54:34.544039 18449 log.cc:824] T 0f79ab7291dd4c1ab96dfc80a1579408 P d910ebb4b7b04c56b7b4c2643f0ae48a: Log is configured to *not* fsync() on all Append() calls
I20260926 20:54:34.544941 18449 tablet_bootstrap.cc:492] T 0f79ab7291dd4c1ab96dfc80a1579408 P d910ebb4b7b04c56b7b4c2643f0ae48a: No bootstrap required, opened a new log
I20260926 20:54:34.545019 18449 ts_tablet_manager.cc:1397] T 0f79ab7291dd4c1ab96dfc80a1579408 P d910ebb4b7b04c56b7b4c2643f0ae48a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:54:34.545300 18447 raft_consensus.cc:348] T 0f79ab7291dd4c1ab96dfc80a1579408 P 05cf556da4b74facb5d2aa28b75d4dd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05cf556da4b74facb5d2aa28b75d4dd8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35347 } } peers { permanent_uuid: "30cf8f26f73c4217a6a52f8586265ef2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41937 } } peers { permanent_uuid: "d910ebb4b7b04c56b7b4c2643f0ae48a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34421 } }
I20260926 20:54:34.545428 18447 raft_consensus.cc:374] T 0f79ab7291dd4c1ab96dfc80a1579408 P 05cf556da4b74facb5d2aa28b75d4dd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:54:34.545449 18447 raft_consensus.cc:729] T 0f79ab7291dd4c1ab96dfc80a1579408 P 05cf556da4b74facb5d2aa28b75d4dd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05cf556da4b74facb5d2aa28b75d4dd8, State: Initialized, Role: FOLLOWER
I20260926 20:54:34.545414 18446 raft_consensus.cc:348] T 0f79ab7291dd4c1ab96dfc80a1579408 P 30cf8f26f73c4217a6a52f8586265ef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05cf556da4b74facb5d2aa28b75d4dd8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35347 } } peers { permanent_uuid: "30cf8f26f73c4217a6a52f8586265ef2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41937 } } peers { permanent_uuid: "d910ebb4b7b04c56b7b4c2643f0ae48a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34421 } }
I20260926 20:54:34.545578 18446 raft_consensus.cc:374] T 0f79ab7291dd4c1ab96dfc80a1579408 P 30cf8f26f73c4217a6a52f8586265ef2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:54:34.545559 18447 consensus_queue.cc:260] T 0f79ab7291dd4c1ab96dfc80a1579408 P 05cf556da4b74facb5d2aa28b75d4dd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05cf556da4b74facb5d2aa28b75d4dd8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35347 } } peers { permanent_uuid: "30cf8f26f73c4217a6a52f8586265ef2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41937 } } peers { permanent_uuid: "d910ebb4b7b04c56b7b4c2643f0ae48a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34421 } }
I20260926 20:54:34.545609 18446 raft_consensus.cc:729] T 0f79ab7291dd4c1ab96dfc80a1579408 P 30cf8f26f73c4217a6a52f8586265ef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30cf8f26f73c4217a6a52f8586265ef2, State: Initialized, Role: FOLLOWER
I20260926 20:54:34.545738 18446 consensus_queue.cc:260] T 0f79ab7291dd4c1ab96dfc80a1579408 P 30cf8f26f73c4217a6a52f8586265ef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05cf556da4b74facb5d2aa28b75d4dd8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35347 } } peers { permanent_uuid: "30cf8f26f73c4217a6a52f8586265ef2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41937 } } peers { permanent_uuid: "d910ebb4b7b04c56b7b4c2643f0ae48a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34421 } }
I20260926 20:54:34.545831 18447 ts_tablet_manager.cc:1428] T 0f79ab7291dd4c1ab96dfc80a1579408 P 05cf556da4b74facb5d2aa28b75d4dd8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 20:54:34.545964 18164 heartbeater.cc:503] Master 127.0.98.62:34565 was elected leader, sending a full tablet report...
I20260926 20:54:34.546000 18446 ts_tablet_manager.cc:1428] T 0f79ab7291dd4c1ab96dfc80a1579408 P 30cf8f26f73c4217a6a52f8586265ef2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:54:34.546149 18298 heartbeater.cc:503] Master 127.0.98.62:34565 was elected leader, sending a full tablet report...
I20260926 20:54:34.546950 18449 raft_consensus.cc:348] T 0f79ab7291dd4c1ab96dfc80a1579408 P d910ebb4b7b04c56b7b4c2643f0ae48a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05cf556da4b74facb5d2aa28b75d4dd8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35347 } } peers { permanent_uuid: "30cf8f26f73c4217a6a52f8586265ef2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41937 } } peers { permanent_uuid: "d910ebb4b7b04c56b7b4c2643f0ae48a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34421 } }
I20260926 20:54:34.547070 18449 raft_consensus.cc:374] T 0f79ab7291dd4c1ab96dfc80a1579408 P d910ebb4b7b04c56b7b4c2643f0ae48a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:54:34.547091 18449 raft_consensus.cc:729] T 0f79ab7291dd4c1ab96dfc80a1579408 P d910ebb4b7b04c56b7b4c2643f0ae48a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d910ebb4b7b04c56b7b4c2643f0ae48a, State: Initialized, Role: FOLLOWER
I20260926 20:54:34.547206 18449 consensus_queue.cc:260] T 0f79ab7291dd4c1ab96dfc80a1579408 P d910ebb4b7b04c56b7b4c2643f0ae48a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05cf556da4b74facb5d2aa28b75d4dd8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35347 } } peers { permanent_uuid: "30cf8f26f73c4217a6a52f8586265ef2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41937 } } peers { permanent_uuid: "d910ebb4b7b04c56b7b4c2643f0ae48a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34421 } }
I20260926 20:54:34.547446 18449 ts_tablet_manager.cc:1428] T 0f79ab7291dd4c1ab96dfc80a1579408 P d910ebb4b7b04c56b7b4c2643f0ae48a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:54:34.547482 18432 heartbeater.cc:503] Master 127.0.98.62:34565 was elected leader, sending a full tablet report...
I20260926 20:54:34.577577 18454 raft_consensus.cc:482] T 0f79ab7291dd4c1ab96dfc80a1579408 P d910ebb4b7b04c56b7b4c2643f0ae48a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:54:34.577694 18454 raft_consensus.cc:504] T 0f79ab7291dd4c1ab96dfc80a1579408 P d910ebb4b7b04c56b7b4c2643f0ae48a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05cf556da4b74facb5d2aa28b75d4dd8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35347 } } peers { permanent_uuid: "30cf8f26f73c4217a6a52f8586265ef2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41937 } } peers { permanent_uuid: "d910ebb4b7b04c56b7b4c2643f0ae48a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34421 } }
I20260926 20:54:34.578153 18454 leader_election.cc:288] T 0f79ab7291dd4c1ab96dfc80a1579408 P d910ebb4b7b04c56b7b4c2643f0ae48a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05cf556da4b74facb5d2aa28b75d4dd8 (127.0.98.1:35347), 30cf8f26f73c4217a6a52f8586265ef2 (127.0.98.2:41937)
I20260926 20:54:34.582376 18443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:34.578440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48691 (local address 127.0.98.2:41937)
0926 20:54:34.578532 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:34.578534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:34.578696 (+ 162us) server_negotiation.cc:407] Connection header received
0926 20:54:34.578806 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:34.578806 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:34.578817 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:34.578892 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:54:34.579013 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:34.579687 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:34.581365 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:34.581960 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:34.582034 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:34.582045 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:34.582254 (+ 209us) server_negotiation.cc:299] Negotiation successful
0926 20:54:34.582298 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 20:54:34.582376 18445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:34.578383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37971 (local address 127.0.98.1:35347)
0926 20:54:34.578426 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:34.578429 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:34.578438 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:54:34.578586 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:34.578589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:34.578605 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:34.578661 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:54:34.578863 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:34.579656 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:34.581366 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:34.581950 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:34.582034 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:34.582045 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:34.582254 (+ 209us) server_negotiation.cc:299] Negotiation successful
0926 20:54:34.582305 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 20:54:34.582409 18434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:34.578305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35347 (local address 127.0.98.3:37971)
0926 20:54:34.578363 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:34.578377 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:34.578435 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:34.578716 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:34.578717 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:34.578719 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:54:34.578789 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:34.578791 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:34.579804 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:34.579805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:34.581226 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:34.581230 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:34.582055 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:34.582056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:34.582182 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:34.582184 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 20:54:34.582238 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 20:54:34.582296 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":84}
I20260926 20:54:34.582417 18455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:34.578358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41937 (local address 127.0.98.3:48691)
0926 20:54:34.578583 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:34.578594 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:34.578721 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:34.578909 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:34.578910 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:34.578911 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:54:34.578958 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:34.578959 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:34.579822 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:34.579823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:34.581226 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:34.581230 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:34.582055 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:34.582056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:34.582168 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:34.582173 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:54:34.582239 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 20:54:34.582295 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":63,"thread_start_us":122,"threads_started":1}
I20260926 20:54:34.582762 18118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f79ab7291dd4c1ab96dfc80a1579408" candidate_uuid: "d910ebb4b7b04c56b7b4c2643f0ae48a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05cf556da4b74facb5d2aa28b75d4dd8" is_pre_election: true
I20260926 20:54:34.582851 18252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f79ab7291dd4c1ab96dfc80a1579408" candidate_uuid: "d910ebb4b7b04c56b7b4c2643f0ae48a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30cf8f26f73c4217a6a52f8586265ef2" is_pre_election: true
I20260926 20:54:34.582962 18118 raft_consensus.cc:2454] T 0f79ab7291dd4c1ab96dfc80a1579408 P 05cf556da4b74facb5d2aa28b75d4dd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d910ebb4b7b04c56b7b4c2643f0ae48a in term 0.
I20260926 20:54:34.583017 18252 raft_consensus.cc:2454] T 0f79ab7291dd4c1ab96dfc80a1579408 P 30cf8f26f73c4217a6a52f8586265ef2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d910ebb4b7b04c56b7b4c2643f0ae48a in term 0.
I20260926 20:54:34.583225 18320 leader_election.cc:302] T 0f79ab7291dd4c1ab96dfc80a1579408 P d910ebb4b7b04c56b7b4c2643f0ae48a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05cf556da4b74facb5d2aa28b75d4dd8, d910ebb4b7b04c56b7b4c2643f0ae48a; no voters:
I20260926 20:54:34.583370 18454 raft_consensus.cc:2781] T 0f79ab7291dd4c1ab96dfc80a1579408 P d910ebb4b7b04c56b7b4c2643f0ae48a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:54:34.583444 18454 raft_consensus.cc:482] T 0f79ab7291dd4c1ab96dfc80a1579408 P d910ebb4b7b04c56b7b4c2643f0ae48a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:54:34.583480 18454 raft_consensus.cc:3037] T 0f79ab7291dd4c1ab96dfc80a1579408 P d910ebb4b7b04c56b7b4c2643f0ae48a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:54:34.584194 18454 raft_consensus.cc:504] T 0f79ab7291dd4c1ab96dfc80a1579408 P d910ebb4b7b04c56b7b4c2643f0ae48a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05cf556da4b74facb5d2aa28b75d4dd8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35347 } } peers { permanent_uuid: "30cf8f26f73c4217a6a52f8586265ef2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41937 } } peers { permanent_uuid: "d910ebb4b7b04c56b7b4c2643f0ae48a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34421 } }
I20260926 20:54:34.584347 18454 leader_election.cc:288] T 0f79ab7291dd4c1ab96dfc80a1579408 P d910ebb4b7b04c56b7b4c2643f0ae48a [CANDIDATE]: Term 1 election: Requested vote from peers 05cf556da4b74facb5d2aa28b75d4dd8 (127.0.98.1:35347), 30cf8f26f73c4217a6a52f8586265ef2 (127.0.98.2:41937)
I20260926 20:54:34.584506 18118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f79ab7291dd4c1ab96dfc80a1579408" candidate_uuid: "d910ebb4b7b04c56b7b4c2643f0ae48a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05cf556da4b74facb5d2aa28b75d4dd8"
I20260926 20:54:34.584551 18252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f79ab7291dd4c1ab96dfc80a1579408" candidate_uuid: "d910ebb4b7b04c56b7b4c2643f0ae48a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30cf8f26f73c4217a6a52f8586265ef2"
I20260926 20:54:34.584595 18118 raft_consensus.cc:3037] T 0f79ab7291dd4c1ab96dfc80a1579408 P 05cf556da4b74facb5d2aa28b75d4dd8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:54:34.584646 18252 raft_consensus.cc:3037] T 0f79ab7291dd4c1ab96dfc80a1579408 P 30cf8f26f73c4217a6a52f8586265ef2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:54:34.585348 18118 raft_consensus.cc:2454] T 0f79ab7291dd4c1ab96dfc80a1579408 P 05cf556da4b74facb5d2aa28b75d4dd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d910ebb4b7b04c56b7b4c2643f0ae48a in term 1.
I20260926 20:54:34.585376 18252 raft_consensus.cc:2454] T 0f79ab7291dd4c1ab96dfc80a1579408 P 30cf8f26f73c4217a6a52f8586265ef2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d910ebb4b7b04c56b7b4c2643f0ae48a in term 1.
I20260926 20:54:34.585513 18320 leader_election.cc:302] T 0f79ab7291dd4c1ab96dfc80a1579408 P d910ebb4b7b04c56b7b4c2643f0ae48a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05cf556da4b74facb5d2aa28b75d4dd8, d910ebb4b7b04c56b7b4c2643f0ae48a; no voters:
I20260926 20:54:34.585602 18454 raft_consensus.cc:2781] T 0f79ab7291dd4c1ab96dfc80a1579408 P d910ebb4b7b04c56b7b4c2643f0ae48a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:54:34.585783 18454 raft_consensus.cc:686] T 0f79ab7291dd4c1ab96dfc80a1579408 P d910ebb4b7b04c56b7b4c2643f0ae48a [term 1 LEADER]: Becoming Leader. State: Replica: d910ebb4b7b04c56b7b4c2643f0ae48a, State: Running, Role: LEADER
I20260926 20:54:34.585896 18454 consensus_queue.cc:237] T 0f79ab7291dd4c1ab96dfc80a1579408 P d910ebb4b7b04c56b7b4c2643f0ae48a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05cf556da4b74facb5d2aa28b75d4dd8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35347 } } peers { permanent_uuid: "30cf8f26f73c4217a6a52f8586265ef2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41937 } } peers { permanent_uuid: "d910ebb4b7b04c56b7b4c2643f0ae48a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34421 } }
I20260926 20:54:34.586753 17972 catalog_manager.cc:5495] T 0f79ab7291dd4c1ab96dfc80a1579408 P d910ebb4b7b04c56b7b4c2643f0ae48a reported cstate change: term changed from 0 to 1, leader changed from <none> to d910ebb4b7b04c56b7b4c2643f0ae48a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "d910ebb4b7b04c56b7b4c2643f0ae48a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05cf556da4b74facb5d2aa28b75d4dd8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30cf8f26f73c4217a6a52f8586265ef2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d910ebb4b7b04c56b7b4c2643f0ae48a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34421 } health_report { overall_health: HEALTHY } } }
I20260926 20:54:34.595171 18444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:34.592827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40502 (local address 127.0.98.3:34421)
0926 20:54:34.592887 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:34.592889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:34.592897 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:54:34.592949 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:34.592950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:34.592965 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:34.593024 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:54:34.593178 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:34.593645 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:34.594362 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:34.594543 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:34.594594 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:34.594718 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:54:34.594802 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:54:34.594880 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:34.595023 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 20:54:34.595050 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 20:54:34.597167 18252 raft_consensus.cc:1261] T 0f79ab7291dd4c1ab96dfc80a1579408 P 30cf8f26f73c4217a6a52f8586265ef2 [term 1 FOLLOWER]: Refusing update from remote peer d910ebb4b7b04c56b7b4c2643f0ae48a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:54:34.597378 18118 raft_consensus.cc:1261] T 0f79ab7291dd4c1ab96dfc80a1579408 P 05cf556da4b74facb5d2aa28b75d4dd8 [term 1 FOLLOWER]: Refusing update from remote peer d910ebb4b7b04c56b7b4c2643f0ae48a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:54:34.597537 18456 consensus_queue.cc:1035] T 0f79ab7291dd4c1ab96dfc80a1579408 P d910ebb4b7b04c56b7b4c2643f0ae48a [LEADER]: Connected to new peer: Peer: permanent_uuid: "30cf8f26f73c4217a6a52f8586265ef2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:54:34.597728 18454 consensus_queue.cc:1035] T 0f79ab7291dd4c1ab96dfc80a1579408 P d910ebb4b7b04c56b7b4c2643f0ae48a [LEADER]: Connected to new peer: Peer: permanent_uuid: "05cf556da4b74facb5d2aa28b75d4dd8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:54:34.599931 18459 mvcc.cc:205] Tried to move back new op lower bound from 7333708081547255808 to 7333708081504493568. Current Snapshot: MvccSnapshot[applied={T|T < 7333708081547255808}]
I20260926 20:54:34.601014 18462 mvcc.cc:205] Tried to move back new op lower bound from 7333708081547255808 to 7333708081504493568. Current Snapshot: MvccSnapshot[applied={T|T < 7333708081547255808 or (T in {7333708081547255808})}]
I20260926 20:54:34.602205 17973 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:54:34.602295 17973 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:54:34.603025 17973 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d5ac981d319d4420b09bebdb8c1a54aa: Sending DeleteTablet for 3 replicas of tablet 0f79ab7291dd4c1ab96dfc80a1579408
I20260926 20:54:34.603369 18366 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f79ab7291dd4c1ab96dfc80a1579408 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:54:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40490
I20260926 20:54:34.603401 18098 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f79ab7291dd4c1ab96dfc80a1579408 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:54:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34004
I20260926 20:54:34.603605 18468 tablet_replica.cc:330] T 0f79ab7291dd4c1ab96dfc80a1579408 P 05cf556da4b74facb5d2aa28b75d4dd8: stopping tablet replica
I20260926 20:54:34.603700 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 18035
I20260926 20:54:34.604379 18232 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f79ab7291dd4c1ab96dfc80a1579408 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:54:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55650
I20260926 20:54:34.604995 18470 tablet_replica.cc:330] T 0f79ab7291dd4c1ab96dfc80a1579408 P 30cf8f26f73c4217a6a52f8586265ef2: stopping tablet replica
I20260926 20:54:34.605080 18470 raft_consensus.cc:2229] T 0f79ab7291dd4c1ab96dfc80a1579408 P 30cf8f26f73c4217a6a52f8586265ef2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:54:34.605149 18470 raft_consensus.cc:2258] T 0f79ab7291dd4c1ab96dfc80a1579408 P 30cf8f26f73c4217a6a52f8586265ef2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:54:34.605334 18469 tablet_replica.cc:330] T 0f79ab7291dd4c1ab96dfc80a1579408 P d910ebb4b7b04c56b7b4c2643f0ae48a: stopping tablet replica
I20260926 20:54:34.605408 18469 raft_consensus.cc:2229] T 0f79ab7291dd4c1ab96dfc80a1579408 P d910ebb4b7b04c56b7b4c2643f0ae48a [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:54:34.605489 18469 raft_consensus.cc:2258] T 0f79ab7291dd4c1ab96dfc80a1579408 P d910ebb4b7b04c56b7b4c2643f0ae48a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:54:34.605597 18470 ts_tablet_manager.cc:1905] T 0f79ab7291dd4c1ab96dfc80a1579408 P 30cf8f26f73c4217a6a52f8586265ef2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:54:34.605818 18469 ts_tablet_manager.cc:1905] T 0f79ab7291dd4c1ab96dfc80a1579408 P d910ebb4b7b04c56b7b4c2643f0ae48a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:54:34.607524 18469 ts_tablet_manager.cc:1918] T 0f79ab7291dd4c1ab96dfc80a1579408 P d910ebb4b7b04c56b7b4c2643f0ae48a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:54:34.607599 18469 log.cc:1196] T 0f79ab7291dd4c1ab96dfc80a1579408 P d910ebb4b7b04c56b7b4c2643f0ae48a: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455725523895-392-0/minicluster-data/ts-2/wal/wals/0f79ab7291dd4c1ab96dfc80a1579408
I20260926 20:54:34.607977 18469 ts_tablet_manager.cc:1939] T 0f79ab7291dd4c1ab96dfc80a1579408 P d910ebb4b7b04c56b7b4c2643f0ae48a: Deleting consensus metadata
I20260926 20:54:34.608352 17957 catalog_manager.cc:4841] TS d910ebb4b7b04c56b7b4c2643f0ae48a (127.0.98.3:34421): tablet 0f79ab7291dd4c1ab96dfc80a1579408 (table test-table [id=adcdb66c4e3b4937a84cd17d2807d072]) successfully deleted
W20260926 20:54:34.610869 17958 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35347 (error 108)
I20260926 20:54:34.611078 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 18169
I20260926 20:54:34.611675 18442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:34.611218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35347 (local address 127.0.0.1:34014)
0926 20:54:34.611603 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:34.611642 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366}
W20260926 20:54:34.612823 17958 catalog_manager.cc:4584] TS 05cf556da4b74facb5d2aa28b75d4dd8 (127.0.98.1:35347): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0f79ab7291dd4c1ab96dfc80a1579408: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35347: connect: Connection refused (error 111)
I20260926 20:54:34.615669 18442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:34.615557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41937 (local address 127.0.0.1:55658)
0926 20:54:34.615610 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:34.615641 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 20:54:34.615679 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 18303
I20260926 20:54:34.620944 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 17941
2026-09-26T20:54:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1902 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T20:54:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:54:34Z Disabled control of system clock
I20260926 20:54:34.647720 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37673
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37673
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 20:54:34.728298 18475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:54:34.728525 18475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:54:34.728546 18475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:54:34.729724 18475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:54:34.729769 18475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:54:34.729784 18475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:54:34.729796 18475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:54:34.729808 18475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:54:34.731221 18475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37673
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:54:34.731499 18475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:54:34.731863 18475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:54:34.736785 18480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:34.736848 18481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:34.736785 18483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:34.737027 18475 server_base.cc:1034] running on GCE node
I20260926 20:54:34.737282 18475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:54:34.737548 18475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:54:34.738693 18475 hybrid_clock.cc:648] HybridClock initialized: now 1790456074738675 us; error 29 us; skew 500 ppm
I20260926 20:54:34.740008 18475 webserver.cc:460] Webserver started at http://127.0.98.62:40629/ using document root <none> and password file <none>
I20260926 20:54:34.740252 18475 fs_manager.cc:359] Metadata directory not provided
I20260926 20:54:34.740334 18475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:54:34.740476 18475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:54:34.741600 18475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/master-0/data/instance:
uuid: "7955b388b4ac449e9f104453d92183c3"
format_stamp: "Formatted at 2026-09-26 20:54:34 on dist-test-slave-hwlj"
server_key: "c556fdcea277ef2ae69d8d14dfdb878d"
server_key_iv: "7f89f90e30ebdf6476e580f1f77fb96c"
server_key_version: "encryptionkey@0"
I20260926 20:54:34.741961 18475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/master-0/wal/instance:
uuid: "7955b388b4ac449e9f104453d92183c3"
format_stamp: "Formatted at 2026-09-26 20:54:34 on dist-test-slave-hwlj"
server_key: "c556fdcea277ef2ae69d8d14dfdb878d"
server_key_iv: "7f89f90e30ebdf6476e580f1f77fb96c"
server_key_version: "encryptionkey@0"
I20260926 20:54:34.743251 18475 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:54:34.744194 18489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:34.744414 18475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:54:34.744520 18475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "7955b388b4ac449e9f104453d92183c3"
format_stamp: "Formatted at 2026-09-26 20:54:34 on dist-test-slave-hwlj"
server_key: "c556fdcea277ef2ae69d8d14dfdb878d"
server_key_iv: "7f89f90e30ebdf6476e580f1f77fb96c"
server_key_version: "encryptionkey@0"
I20260926 20:54:34.744592 18475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:54:34.754668 18475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:54:34.754974 18475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:54:34.755115 18475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:54:34.759060 18475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37673
I20260926 20:54:34.759097 18541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37673 every 8 connection(s)
I20260926 20:54:34.759454 18475 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:54:34.760448 18542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:54:34.762045 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 18475
I20260926 20:54:34.762154 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/master-0/wal/instance
I20260926 20:54:34.762362 392 external_mini_cluster.cc:1428] Setting key ef7cd7e4885dc500ccb7a73ef5f1ada7
I20260926 20:54:34.762976 18542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7955b388b4ac449e9f104453d92183c3: Bootstrap starting.
I20260926 20:54:34.763695 18542 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7955b388b4ac449e9f104453d92183c3: Neither blocks nor log segments found. Creating new log.
I20260926 20:54:34.764056 18542 log.cc:824] T 00000000000000000000000000000000 P 7955b388b4ac449e9f104453d92183c3: Log is configured to *not* fsync() on all Append() calls
I20260926 20:54:34.764792 18542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7955b388b4ac449e9f104453d92183c3: No bootstrap required, opened a new log
I20260926 20:54:34.766989 18542 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7955b388b4ac449e9f104453d92183c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7955b388b4ac449e9f104453d92183c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37673 } }
I20260926 20:54:34.767171 18542 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7955b388b4ac449e9f104453d92183c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:54:34.767208 18542 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7955b388b4ac449e9f104453d92183c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7955b388b4ac449e9f104453d92183c3, State: Initialized, Role: FOLLOWER
I20260926 20:54:34.767377 18542 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7955b388b4ac449e9f104453d92183c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7955b388b4ac449e9f104453d92183c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37673 } }
I20260926 20:54:34.767465 18542 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7955b388b4ac449e9f104453d92183c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:54:34.767499 18542 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7955b388b4ac449e9f104453d92183c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:54:34.767545 18542 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7955b388b4ac449e9f104453d92183c3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:54:34.768321 18542 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7955b388b4ac449e9f104453d92183c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7955b388b4ac449e9f104453d92183c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37673 } }
I20260926 20:54:34.768433 18542 leader_election.cc:302] T 00000000000000000000000000000000 P 7955b388b4ac449e9f104453d92183c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7955b388b4ac449e9f104453d92183c3; no voters:
I20260926 20:54:34.768602 18542 leader_election.cc:288] T 00000000000000000000000000000000 P 7955b388b4ac449e9f104453d92183c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:54:34.768662 18547 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7955b388b4ac449e9f104453d92183c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:54:34.768851 18547 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7955b388b4ac449e9f104453d92183c3 [term 1 LEADER]: Becoming Leader. State: Replica: 7955b388b4ac449e9f104453d92183c3, State: Running, Role: LEADER
I20260926 20:54:34.768905 18545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:34.762893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60838 (local address 127.0.98.62:37673)
0926 20:54:34.763110 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:34.763119 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:34.763150 (+ 31us) server_negotiation.cc:407] Connection header received
0926 20:54:34.763737 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:34.763775 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:34.763874 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:34.764034 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:34.764248 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:34.765596 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:34.766372 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:34.767069 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:34.767187 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:34.768057 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:34.768062 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:34.768062 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:54:34.768273 (+ 211us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:34.768319 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:54:34.768335 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:34.768408 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:34.768495 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 20:54:34.768693 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":85,"threads_started":1}
I20260926 20:54:34.768877 18542 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7955b388b4ac449e9f104453d92183c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:54:34.768992 18547 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7955b388b4ac449e9f104453d92183c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7955b388b4ac449e9f104453d92183c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37673 } }
I20260926 20:54:34.770126 18549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7955b388b4ac449e9f104453d92183c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7955b388b4ac449e9f104453d92183c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7955b388b4ac449e9f104453d92183c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37673 } } }
I20260926 20:54:34.770212 18549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7955b388b4ac449e9f104453d92183c3 [sys.catalog]: This master's current role is: LEADER
I20260926 20:54:34.770512 18548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7955b388b4ac449e9f104453d92183c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7955b388b4ac449e9f104453d92183c3. Latest consensus state: current_term: 1 leader_uuid: "7955b388b4ac449e9f104453d92183c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7955b388b4ac449e9f104453d92183c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37673 } } }
I20260926 20:54:34.770602 18548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7955b388b4ac449e9f104453d92183c3 [sys.catalog]: This master's current role is: LEADER
W20260926 20:54:34.770852 18563 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7955b388b4ac449e9f104453d92183c3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:54:34.770890 18563 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:54:34.770954 18562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:54:34.771137 18562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:54:34.772435 18562 catalog_manager.cc:1347] Generated new cluster ID: b285da5428c84619bb9287d997835d49
I20260926 20:54:34.772491 18562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:54:34.777853 18562 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:54:34.778369 18562 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:54:34.781287 18562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7955b388b4ac449e9f104453d92183c3: Generated new TSK 0
I20260926 20:54:34.781466 18562 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:54:34.783334 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37673
--builtin_ntp_servers=127.0.98.20:43093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 20:54:34.871862 18566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:54:34.872084 18566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:54:34.872121 18566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:54:34.873281 18566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:54:34.873339 18566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:54:34.873379 18566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:54:34.874987 18566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:54:34.875257 18566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:54:34.875551 18566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:54:34.881481 18574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:34.881484 18572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:34.881844 18566 server_base.cc:1034] running on GCE node
W20260926 20:54:34.881570 18571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:34.882112 18566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:54:34.882421 18566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:54:34.883610 18566 hybrid_clock.cc:648] HybridClock initialized: now 1790456074883595 us; error 28 us; skew 500 ppm
I20260926 20:54:34.885210 18566 webserver.cc:460] Webserver started at http://127.0.98.1:35767/ using document root <none> and password file <none>
I20260926 20:54:34.885496 18566 fs_manager.cc:359] Metadata directory not provided
I20260926 20:54:34.885566 18566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:54:34.885703 18566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:54:34.887115 18566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/ts-0/data/instance:
uuid: "9ed41cc926134f55b4e8216d60f3290a"
format_stamp: "Formatted at 2026-09-26 20:54:34 on dist-test-slave-hwlj"
server_key: "5e1be15e86fab107a54e5612005351a7"
server_key_iv: "c2ffa29fb602377742bde72c8112c18c"
server_key_version: "encryptionkey@0"
I20260926 20:54:34.887508 18566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9ed41cc926134f55b4e8216d60f3290a"
format_stamp: "Formatted at 2026-09-26 20:54:34 on dist-test-slave-hwlj"
server_key: "5e1be15e86fab107a54e5612005351a7"
server_key_iv: "c2ffa29fb602377742bde72c8112c18c"
server_key_version: "encryptionkey@0"
I20260926 20:54:34.889117 18566 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260926 20:54:34.889940 18580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:34.890101 18566 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 20:54:34.890192 18566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/ts-0/wal
uuid: "9ed41cc926134f55b4e8216d60f3290a"
format_stamp: "Formatted at 2026-09-26 20:54:34 on dist-test-slave-hwlj"
server_key: "5e1be15e86fab107a54e5612005351a7"
server_key_iv: "c2ffa29fb602377742bde72c8112c18c"
server_key_version: "encryptionkey@0"
I20260926 20:54:34.890257 18566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:54:34.906759 18566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:54:34.907075 18566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:54:34.907217 18566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:54:34.907450 18566 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:54:34.907840 18566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:54:34.907876 18566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:34.907898 18566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:54:34.907935 18566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:34.913445 18566 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35175
I20260926 20:54:34.913527 18693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35175 every 8 connection(s)
I20260926 20:54:34.913861 18566 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:54:34.918907 18545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:34.914620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50639 (local address 127.0.98.62:37673)
0926 20:54:34.914694 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:34.914696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:34.914937 (+ 241us) server_negotiation.cc:407] Connection header received
0926 20:54:34.915417 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:34.915418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:34.915430 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:34.915489 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:34.916225 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:34.916715 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:34.917671 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:34.917859 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:34.917911 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:34.918658 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:34.918659 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:34.918659 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:54:34.918663 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:34.918673 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:54:34.918676 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:34.918714 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:34.918822 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 20:54:34.918846 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 20:54:34.919056 18696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:34.914608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.1:50639)
0926 20:54:34.914807 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:34.914849 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:34.915330 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:34.915549 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:34.915552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:34.915560 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:54:34.916135 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:34.916141 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:34.916798 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:34.916799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:34.917599 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:34.917602 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:34.917930 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:34.917930 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:34.918035 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:34.918541 (+ 506us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:54:34.918542 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:54:34.918570 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:54:34.918592 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:54:34.918720 (+ 128us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:54:34.918722 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:54:34.918725 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:54:34.918788 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 20:54:34.918864 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":94,"threads_started":1}
I20260926 20:54:34.919581 18694 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37673
I20260926 20:54:34.919689 18694 heartbeater.cc:464] Registering TS with master...
I20260926 20:54:34.919885 18694 heartbeater.cc:511] Master 127.0.98.62:37673 requested a full tablet report, sending...
I20260926 20:54:34.920394 18506 ts_manager.cc:194] Registered new tserver with Master: 9ed41cc926134f55b4e8216d60f3290a (127.0.98.1:35175)
I20260926 20:54:34.920734 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 18566
I20260926 20:54:34.920842 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/ts-0/wal/instance
I20260926 20:54:34.921015 392 external_mini_cluster.cc:1428] Setting key 7431cb74acd09b2d8f647c382a797b8d
I20260926 20:54:34.921151 18506 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:50639
I20260926 20:54:34.921913 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37673
--builtin_ntp_servers=127.0.98.20:43093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 20:54:35.002789 18697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:54:35.003014 18697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:54:35.003051 18697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:54:35.004492 18697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:54:35.004552 18697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:54:35.004597 18697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:54:35.005942 18697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:54:35.006212 18697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:54:35.006505 18697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:54:35.011830 18702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:35.011972 18703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:35.011840 18705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:35.012094 18697 server_base.cc:1034] running on GCE node
I20260926 20:54:35.012308 18697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:54:35.012555 18697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:54:35.013715 18697 hybrid_clock.cc:648] HybridClock initialized: now 1790456075013698 us; error 30 us; skew 500 ppm
I20260926 20:54:35.014953 18697 webserver.cc:460] Webserver started at http://127.0.98.2:39683/ using document root <none> and password file <none>
I20260926 20:54:35.015221 18697 fs_manager.cc:359] Metadata directory not provided
I20260926 20:54:35.015309 18697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:54:35.015466 18697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:54:35.016672 18697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/ts-1/data/instance:
uuid: "e4427d79675f4e0b91f3b6cd5d8358b5"
format_stamp: "Formatted at 2026-09-26 20:54:35 on dist-test-slave-hwlj"
server_key: "de3b642ff00926d65f76fe5d0a4576dc"
server_key_iv: "73926ccc14598b7f0dbb2c0d1100406b"
server_key_version: "encryptionkey@0"
I20260926 20:54:35.017053 18697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e4427d79675f4e0b91f3b6cd5d8358b5"
format_stamp: "Formatted at 2026-09-26 20:54:35 on dist-test-slave-hwlj"
server_key: "de3b642ff00926d65f76fe5d0a4576dc"
server_key_iv: "73926ccc14598b7f0dbb2c0d1100406b"
server_key_version: "encryptionkey@0"
I20260926 20:54:35.018399 18697 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 20:54:35.019129 18711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:35.019305 18697 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 20:54:35.019393 18697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/ts-1/wal
uuid: "e4427d79675f4e0b91f3b6cd5d8358b5"
format_stamp: "Formatted at 2026-09-26 20:54:35 on dist-test-slave-hwlj"
server_key: "de3b642ff00926d65f76fe5d0a4576dc"
server_key_iv: "73926ccc14598b7f0dbb2c0d1100406b"
server_key_version: "encryptionkey@0"
I20260926 20:54:35.019471 18697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:54:35.027428 18697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:54:35.027714 18697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:54:35.027877 18697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:54:35.028096 18697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:54:35.028419 18697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:54:35.028467 18697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:35.028506 18697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:54:35.028534 18697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:35.034287 18697 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46499
I20260926 20:54:35.034328 18824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46499 every 8 connection(s)
I20260926 20:54:35.034642 18697 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
I20260926 20:54:35.036310 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 18697
I20260926 20:54:35.036408 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/ts-1/wal/instance
I20260926 20:54:35.036616 392 external_mini_cluster.cc:1428] Setting key f4114e05da230cfc755cd477206f5cf6
I20260926 20:54:35.037626 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37673
--builtin_ntp_servers=127.0.98.20:43093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 20:54:35.040179 18545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:35.035285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37449 (local address 127.0.98.62:37673)
0926 20:54:35.035364 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:35.035368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:35.035572 (+ 204us) server_negotiation.cc:407] Connection header received
0926 20:54:35.036096 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:35.036097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:35.036107 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:35.036152 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:35.037074 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:35.037687 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:35.038712 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:35.038915 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:35.038990 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:35.039899 (+ 909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:35.039901 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:35.039901 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:54:35.039910 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:35.039924 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:54:35.039927 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:35.039982 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:35.040072 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 20:54:35.040099 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 20:54:35.040398 18827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:35.035260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.2:37449)
0926 20:54:35.035460 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:35.035493 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:35.036013 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:35.036233 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:35.036238 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:35.036256 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:54:35.036980 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:35.036988 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:35.037839 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:35.037840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:35.038619 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:35.038621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:35.038997 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:35.038998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:35.039132 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:35.039698 (+ 566us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:54:35.039698 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:54:35.039726 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:54:35.039802 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:54:35.039983 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:54:35.039989 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:54:35.039993 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:54:35.040076 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 20:54:35.040179 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":119,"threads_started":1}
I20260926 20:54:35.040951 18825 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37673
I20260926 20:54:35.041051 18825 heartbeater.cc:464] Registering TS with master...
I20260926 20:54:35.041226 18825 heartbeater.cc:511] Master 127.0.98.62:37673 requested a full tablet report, sending...
I20260926 20:54:35.041591 18506 ts_manager.cc:194] Registered new tserver with Master: e4427d79675f4e0b91f3b6cd5d8358b5 (127.0.98.2:46499)
I20260926 20:54:35.041954 18506 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:37449
W20260926 20:54:35.118937 18828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:54:35.119153 18828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:54:35.119176 18828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:54:35.120431 18828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:54:35.120476 18828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:54:35.120501 18828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:54:35.121973 18828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:54:35.122251 18828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:54:35.122565 18828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:54:35.127521 18833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:35.127512 18836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:35.127512 18834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:35.127674 18828 server_base.cc:1034] running on GCE node
I20260926 20:54:35.128000 18828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:54:35.128247 18828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:54:35.129424 18828 hybrid_clock.cc:648] HybridClock initialized: now 1790456075129403 us; error 36 us; skew 500 ppm
I20260926 20:54:35.130807 18828 webserver.cc:460] Webserver started at http://127.0.98.3:38983/ using document root <none> and password file <none>
I20260926 20:54:35.131057 18828 fs_manager.cc:359] Metadata directory not provided
I20260926 20:54:35.131140 18828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:54:35.131281 18828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:54:35.132503 18828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/ts-2/data/instance:
uuid: "26697cf7e0624a09a7c2d456f27629d0"
format_stamp: "Formatted at 2026-09-26 20:54:35 on dist-test-slave-hwlj"
server_key: "09232b9b5a94dd3f265920168452dc4c"
server_key_iv: "cb517921485f0e3c68785cbd3a512c53"
server_key_version: "encryptionkey@0"
I20260926 20:54:35.132867 18828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/ts-2/wal/instance:
uuid: "26697cf7e0624a09a7c2d456f27629d0"
format_stamp: "Formatted at 2026-09-26 20:54:35 on dist-test-slave-hwlj"
server_key: "09232b9b5a94dd3f265920168452dc4c"
server_key_iv: "cb517921485f0e3c68785cbd3a512c53"
server_key_version: "encryptionkey@0"
I20260926 20:54:35.134209 18828 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:54:35.134945 18842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:35.135110 18828 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:54:35.135214 18828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/ts-2/wal
uuid: "26697cf7e0624a09a7c2d456f27629d0"
format_stamp: "Formatted at 2026-09-26 20:54:35 on dist-test-slave-hwlj"
server_key: "09232b9b5a94dd3f265920168452dc4c"
server_key_iv: "cb517921485f0e3c68785cbd3a512c53"
server_key_version: "encryptionkey@0"
I20260926 20:54:35.135285 18828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:54:35.172487 18828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:54:35.172837 18828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:54:35.172950 18828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:54:35.173149 18828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:54:35.173442 18828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:54:35.173472 18828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:35.173494 18828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:54:35.173507 18828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:35.179127 18828 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40533
I20260926 20:54:35.179169 18955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40533 every 8 connection(s)
I20260926 20:54:35.179507 18828 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
I20260926 20:54:35.182891 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 18828
I20260926 20:54:35.182965 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/ts-2/wal/instance
I20260926 20:54:35.183158 392 external_mini_cluster.cc:1428] Setting key 230901b170bef7150c730a3cae78f666
I20260926 20:54:35.184965 18545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:35.180243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60391 (local address 127.0.98.62:37673)
0926 20:54:35.180309 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:35.180311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:35.180532 (+ 221us) server_negotiation.cc:407] Connection header received
0926 20:54:35.181036 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:35.181038 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:35.181048 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:35.181112 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:35.181817 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:35.182294 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:35.183405 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:35.183715 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:35.183844 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:35.184753 (+ 909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:35.184754 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:35.184754 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:54:35.184758 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:35.184773 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:54:35.184777 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:35.184834 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:35.184886 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 20:54:35.184906 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 20:54:35.185220 18958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:35.180227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.3:60391)
0926 20:54:35.180410 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:35.180450 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:35.180927 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:35.181189 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:35.181192 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:35.181201 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:54:35.181746 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:35.181751 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:35.182363 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:35.182363 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:35.183322 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:35.183326 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:35.183856 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:35.183858 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:35.184013 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:35.184576 (+ 563us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:54:35.184577 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:54:35.184606 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:54:35.184638 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:54:35.184837 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:54:35.184839 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:54:35.184842 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:54:35.184887 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 20:54:35.184968 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":90,"threads_started":1}
I20260926 20:54:35.185657 18956 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37673
I20260926 20:54:35.185752 18956 heartbeater.cc:464] Registering TS with master...
I20260926 20:54:35.185897 18956 heartbeater.cc:511] Master 127.0.98.62:37673 requested a full tablet report, sending...
I20260926 20:54:35.186257 18506 ts_manager.cc:194] Registered new tserver with Master: 26697cf7e0624a09a7c2d456f27629d0 (127.0.98.3:40533)
I20260926 20:54:35.186618 18506 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:60391
I20260926 20:54:35.194988 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:54:35.199687 18545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:35.196522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60840 (local address 127.0.98.62:37673)
0926 20:54:35.196602 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:35.196604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:35.196694 (+ 90us) server_negotiation.cc:407] Connection header received
0926 20:54:35.196771 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:35.196772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:35.196784 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:35.196834 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:35.197019 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:35.197513 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:35.198420 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:35.198634 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:35.198707 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:35.199385 (+ 678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:35.199387 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:35.199387 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:54:35.199392 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:35.199406 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:54:35.199410 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:35.199515 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:35.199592 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 20:54:35.199603 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 20:54:35.201373 18506 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60840:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:54:35.202996 18506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:54:35.209623 18965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:35.205416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35175 (local address 127.0.0.1:49904)
0926 20:54:35.205824 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:35.205834 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:35.205890 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:35.206172 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:35.206174 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:35.206176 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:54:35.206267 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:35.206268 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:35.207574 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:35.207575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:35.208638 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:35.208639 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:35.209322 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:35.209323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:35.209396 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:35.209409 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 20:54:35.209483 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 20:54:35.209541 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"mutex_wait_us":26,"thread_start_us":83,"threads_started":1}
I20260926 20:54:35.209632 18969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:35.205647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49904 (local address 127.0.98.1:35175)
0926 20:54:35.205919 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:35.205922 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:35.205932 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:54:35.205991 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:35.205993 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:35.206007 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:35.206085 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:54:35.206348 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:35.207483 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:35.208724 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:35.209216 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:35.209313 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:35.209413 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:35.209501 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 20:54:35.209541 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":88,"threads_started":1}
I20260926 20:54:35.209935 18964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:35.205387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46499 (local address 127.0.0.1:60944)
0926 20:54:35.205578 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:35.205616 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:35.205690 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:35.206247 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:35.206248 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:35.206250 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:54:35.206298 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:35.206300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:35.207187 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:35.207188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:35.208605 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:35.208608 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:35.209652 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:35.209653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:35.209721 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:35.209725 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:54:35.209784 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 20:54:35.209832 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":85,"threads_started":1}
I20260926 20:54:35.209981 18967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:35.205467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60944 (local address 127.0.98.2:46499)
0926 20:54:35.206020 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:35.206025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:35.206035 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:54:35.206074 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:35.206076 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:35.206093 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:35.206169 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:54:35.206389 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:35.207102 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:35.208701 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:35.209456 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:35.209527 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:35.209615 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:35.209819 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 20:54:35.209877 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":78,"threads_started":1}
I20260926 20:54:35.210115 18968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:35.205645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34030 (local address 127.0.98.3:40533)
0926 20:54:35.205828 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:35.205833 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:35.205913 (+ 80us) server_negotiation.cc:407] Connection header received
0926 20:54:35.205978 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:35.205980 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:35.205996 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:35.206085 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:54:35.206348 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:35.207489 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:35.208880 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:35.209549 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:35.209605 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:35.209693 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:35.209975 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 20:54:35.210023 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":85,"threads_started":1}
I20260926 20:54:35.209939 18966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:35.205406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40533 (local address 127.0.0.1:34030)
0926 20:54:35.205829 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:35.205840 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:35.205906 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:35.206170 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:35.206174 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:35.206179 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:54:35.206262 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:35.206264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:35.207840 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:35.207841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:35.208787 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:35.208789 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:35.209723 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:35.209724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:35.209784 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:35.209787 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:54:35.209831 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 20:54:35.209862 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":90,"threads_started":1}
I20260926 20:54:35.210863 18759 tablet_service.cc:1467] Processing CreateTablet for tablet 965bbe21dec14927b1823e97ad4fa5d1 (DEFAULT_TABLE table=test-table [id=5b74c542b55445688ab01a7be388704d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:54:35.210887 18890 tablet_service.cc:1467] Processing CreateTablet for tablet 965bbe21dec14927b1823e97ad4fa5d1 (DEFAULT_TABLE table=test-table [id=5b74c542b55445688ab01a7be388704d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:54:35.211262 18759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 965bbe21dec14927b1823e97ad4fa5d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:54:35.211262 18890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 965bbe21dec14927b1823e97ad4fa5d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:54:35.212641 18628 tablet_service.cc:1467] Processing CreateTablet for tablet 965bbe21dec14927b1823e97ad4fa5d1 (DEFAULT_TABLE table=test-table [id=5b74c542b55445688ab01a7be388704d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:54:35.212992 18628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 965bbe21dec14927b1823e97ad4fa5d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:54:35.214675 18971 tablet_bootstrap.cc:492] T 965bbe21dec14927b1823e97ad4fa5d1 P e4427d79675f4e0b91f3b6cd5d8358b5: Bootstrap starting.
I20260926 20:54:35.214896 18970 tablet_bootstrap.cc:492] T 965bbe21dec14927b1823e97ad4fa5d1 P 26697cf7e0624a09a7c2d456f27629d0: Bootstrap starting.
I20260926 20:54:35.215574 18970 tablet_bootstrap.cc:654] T 965bbe21dec14927b1823e97ad4fa5d1 P 26697cf7e0624a09a7c2d456f27629d0: Neither blocks nor log segments found. Creating new log.
I20260926 20:54:35.215737 18972 tablet_bootstrap.cc:492] T 965bbe21dec14927b1823e97ad4fa5d1 P 9ed41cc926134f55b4e8216d60f3290a: Bootstrap starting.
I20260926 20:54:35.216082 18970 log.cc:824] T 965bbe21dec14927b1823e97ad4fa5d1 P 26697cf7e0624a09a7c2d456f27629d0: Log is configured to *not* fsync() on all Append() calls
I20260926 20:54:35.216562 18971 tablet_bootstrap.cc:654] T 965bbe21dec14927b1823e97ad4fa5d1 P e4427d79675f4e0b91f3b6cd5d8358b5: Neither blocks nor log segments found. Creating new log.
I20260926 20:54:35.216562 18972 tablet_bootstrap.cc:654] T 965bbe21dec14927b1823e97ad4fa5d1 P 9ed41cc926134f55b4e8216d60f3290a: Neither blocks nor log segments found. Creating new log.
I20260926 20:54:35.216804 18970 tablet_bootstrap.cc:492] T 965bbe21dec14927b1823e97ad4fa5d1 P 26697cf7e0624a09a7c2d456f27629d0: No bootstrap required, opened a new log
I20260926 20:54:35.216881 18970 ts_tablet_manager.cc:1397] T 965bbe21dec14927b1823e97ad4fa5d1 P 26697cf7e0624a09a7c2d456f27629d0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:54:35.216892 18972 log.cc:824] T 965bbe21dec14927b1823e97ad4fa5d1 P 9ed41cc926134f55b4e8216d60f3290a: Log is configured to *not* fsync() on all Append() calls
I20260926 20:54:35.216888 18971 log.cc:824] T 965bbe21dec14927b1823e97ad4fa5d1 P e4427d79675f4e0b91f3b6cd5d8358b5: Log is configured to *not* fsync() on all Append() calls
I20260926 20:54:35.217612 18972 tablet_bootstrap.cc:492] T 965bbe21dec14927b1823e97ad4fa5d1 P 9ed41cc926134f55b4e8216d60f3290a: No bootstrap required, opened a new log
I20260926 20:54:35.217685 18972 ts_tablet_manager.cc:1397] T 965bbe21dec14927b1823e97ad4fa5d1 P 9ed41cc926134f55b4e8216d60f3290a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 20:54:35.217890 18971 tablet_bootstrap.cc:492] T 965bbe21dec14927b1823e97ad4fa5d1 P e4427d79675f4e0b91f3b6cd5d8358b5: No bootstrap required, opened a new log
I20260926 20:54:35.217953 18971 ts_tablet_manager.cc:1397] T 965bbe21dec14927b1823e97ad4fa5d1 P e4427d79675f4e0b91f3b6cd5d8358b5: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 20:54:35.219039 18970 raft_consensus.cc:348] T 965bbe21dec14927b1823e97ad4fa5d1 P 26697cf7e0624a09a7c2d456f27629d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26697cf7e0624a09a7c2d456f27629d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40533 } } peers { permanent_uuid: "e4427d79675f4e0b91f3b6cd5d8358b5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46499 } } peers { permanent_uuid: "9ed41cc926134f55b4e8216d60f3290a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35175 } }
I20260926 20:54:35.219201 18970 raft_consensus.cc:374] T 965bbe21dec14927b1823e97ad4fa5d1 P 26697cf7e0624a09a7c2d456f27629d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:54:35.219245 18970 raft_consensus.cc:729] T 965bbe21dec14927b1823e97ad4fa5d1 P 26697cf7e0624a09a7c2d456f27629d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26697cf7e0624a09a7c2d456f27629d0, State: Initialized, Role: FOLLOWER
I20260926 20:54:35.219379 18970 consensus_queue.cc:260] T 965bbe21dec14927b1823e97ad4fa5d1 P 26697cf7e0624a09a7c2d456f27629d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26697cf7e0624a09a7c2d456f27629d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40533 } } peers { permanent_uuid: "e4427d79675f4e0b91f3b6cd5d8358b5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46499 } } peers { permanent_uuid: "9ed41cc926134f55b4e8216d60f3290a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35175 } }
I20260926 20:54:35.219655 18970 ts_tablet_manager.cc:1428] T 965bbe21dec14927b1823e97ad4fa5d1 P 26697cf7e0624a09a7c2d456f27629d0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:54:35.219786 18956 heartbeater.cc:503] Master 127.0.98.62:37673 was elected leader, sending a full tablet report...
I20260926 20:54:35.220322 18972 raft_consensus.cc:348] T 965bbe21dec14927b1823e97ad4fa5d1 P 9ed41cc926134f55b4e8216d60f3290a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26697cf7e0624a09a7c2d456f27629d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40533 } } peers { permanent_uuid: "e4427d79675f4e0b91f3b6cd5d8358b5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46499 } } peers { permanent_uuid: "9ed41cc926134f55b4e8216d60f3290a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35175 } }
I20260926 20:54:35.220482 18972 raft_consensus.cc:374] T 965bbe21dec14927b1823e97ad4fa5d1 P 9ed41cc926134f55b4e8216d60f3290a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:54:35.220526 18972 raft_consensus.cc:729] T 965bbe21dec14927b1823e97ad4fa5d1 P 9ed41cc926134f55b4e8216d60f3290a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ed41cc926134f55b4e8216d60f3290a, State: Initialized, Role: FOLLOWER
I20260926 20:54:35.220525 18971 raft_consensus.cc:348] T 965bbe21dec14927b1823e97ad4fa5d1 P e4427d79675f4e0b91f3b6cd5d8358b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26697cf7e0624a09a7c2d456f27629d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40533 } } peers { permanent_uuid: "e4427d79675f4e0b91f3b6cd5d8358b5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46499 } } peers { permanent_uuid: "9ed41cc926134f55b4e8216d60f3290a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35175 } }
I20260926 20:54:35.220665 18971 raft_consensus.cc:374] T 965bbe21dec14927b1823e97ad4fa5d1 P e4427d79675f4e0b91f3b6cd5d8358b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:54:35.220642 18972 consensus_queue.cc:260] T 965bbe21dec14927b1823e97ad4fa5d1 P 9ed41cc926134f55b4e8216d60f3290a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26697cf7e0624a09a7c2d456f27629d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40533 } } peers { permanent_uuid: "e4427d79675f4e0b91f3b6cd5d8358b5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46499 } } peers { permanent_uuid: "9ed41cc926134f55b4e8216d60f3290a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35175 } }
I20260926 20:54:35.220710 18971 raft_consensus.cc:729] T 965bbe21dec14927b1823e97ad4fa5d1 P e4427d79675f4e0b91f3b6cd5d8358b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4427d79675f4e0b91f3b6cd5d8358b5, State: Initialized, Role: FOLLOWER
I20260926 20:54:35.220855 18971 consensus_queue.cc:260] T 965bbe21dec14927b1823e97ad4fa5d1 P e4427d79675f4e0b91f3b6cd5d8358b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26697cf7e0624a09a7c2d456f27629d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40533 } } peers { permanent_uuid: "e4427d79675f4e0b91f3b6cd5d8358b5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46499 } } peers { permanent_uuid: "9ed41cc926134f55b4e8216d60f3290a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35175 } }
I20260926 20:54:35.220901 18972 ts_tablet_manager.cc:1428] T 965bbe21dec14927b1823e97ad4fa5d1 P 9ed41cc926134f55b4e8216d60f3290a: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260926 20:54:35.221030 18694 heartbeater.cc:503] Master 127.0.98.62:37673 was elected leader, sending a full tablet report...
I20260926 20:54:35.221128 18971 ts_tablet_manager.cc:1428] T 965bbe21dec14927b1823e97ad4fa5d1 P e4427d79675f4e0b91f3b6cd5d8358b5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:54:35.221150 18825 heartbeater.cc:503] Master 127.0.98.62:37673 was elected leader, sending a full tablet report...
W20260926 20:54:35.285147 18826 tablet.cc:2369] T 965bbe21dec14927b1823e97ad4fa5d1 P e4427d79675f4e0b91f3b6cd5d8358b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:54:35.343180 18977 raft_consensus.cc:482] T 965bbe21dec14927b1823e97ad4fa5d1 P 9ed41cc926134f55b4e8216d60f3290a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:54:35.343307 18977 raft_consensus.cc:504] T 965bbe21dec14927b1823e97ad4fa5d1 P 9ed41cc926134f55b4e8216d60f3290a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26697cf7e0624a09a7c2d456f27629d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40533 } } peers { permanent_uuid: "e4427d79675f4e0b91f3b6cd5d8358b5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46499 } } peers { permanent_uuid: "9ed41cc926134f55b4e8216d60f3290a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35175 } }
I20260926 20:54:35.343724 18977 leader_election.cc:288] T 965bbe21dec14927b1823e97ad4fa5d1 P 9ed41cc926134f55b4e8216d60f3290a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26697cf7e0624a09a7c2d456f27629d0 (127.0.98.3:40533), e4427d79675f4e0b91f3b6cd5d8358b5 (127.0.98.2:46499)
I20260926 20:54:35.346647 18967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:35.344050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41053 (local address 127.0.98.2:46499)
0926 20:54:35.344094 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:35.344096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:35.344106 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:54:35.344133 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:35.344134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:35.344145 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:35.344237 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:54:35.344426 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:35.344998 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:35.345955 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:35.346317 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:35.346390 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:35.346401 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:35.346529 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 20:54:35.346572 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 20:54:35.346653 18696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:35.343934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46499 (local address 127.0.98.1:41053)
0926 20:54:35.344004 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:35.344016 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:35.344061 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:35.344266 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:35.344267 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:35.344269 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:54:35.344352 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:35.344354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:35.345074 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:35.345075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:35.345872 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:35.345874 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:35.346389 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:35.346390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:35.346460 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:35.346463 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:54:35.346511 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 20:54:35.346567 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":17}
I20260926 20:54:35.346663 18968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:35.344120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59693 (local address 127.0.98.3:40533)
0926 20:54:35.344178 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:35.344180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:35.344297 (+ 117us) server_negotiation.cc:407] Connection header received
0926 20:54:35.344385 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:35.344386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:35.344397 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:35.344454 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:54:35.344594 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:35.345075 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:35.345985 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:35.346322 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:35.346400 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:35.346409 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:35.346568 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 20:54:35.346606 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 20:54:35.346683 18979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:35.343959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40533 (local address 127.0.98.1:59693)
0926 20:54:35.344242 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:35.344252 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:35.344321 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:35.344466 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:35.344467 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:35.344469 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:54:35.344531 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:35.344532 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:35.345150 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:35.345151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:35.345883 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:35.345884 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:35.346411 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:35.346412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:35.346483 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:35.346486 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:54:35.346555 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 20:54:35.346589 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":1,"thread_start_us":92,"threads_started":1}
I20260926 20:54:35.347007 18779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "965bbe21dec14927b1823e97ad4fa5d1" candidate_uuid: "9ed41cc926134f55b4e8216d60f3290a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4427d79675f4e0b91f3b6cd5d8358b5" is_pre_election: true
I20260926 20:54:35.347198 18779 raft_consensus.cc:2454] T 965bbe21dec14927b1823e97ad4fa5d1 P e4427d79675f4e0b91f3b6cd5d8358b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ed41cc926134f55b4e8216d60f3290a in term 0.
I20260926 20:54:35.347371 18910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "965bbe21dec14927b1823e97ad4fa5d1" candidate_uuid: "9ed41cc926134f55b4e8216d60f3290a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26697cf7e0624a09a7c2d456f27629d0" is_pre_election: true
I20260926 20:54:35.347458 18584 leader_election.cc:302] T 965bbe21dec14927b1823e97ad4fa5d1 P 9ed41cc926134f55b4e8216d60f3290a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ed41cc926134f55b4e8216d60f3290a, e4427d79675f4e0b91f3b6cd5d8358b5; no voters:
I20260926 20:54:35.347533 18910 raft_consensus.cc:2454] T 965bbe21dec14927b1823e97ad4fa5d1 P 26697cf7e0624a09a7c2d456f27629d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ed41cc926134f55b4e8216d60f3290a in term 0.
I20260926 20:54:35.347589 18977 raft_consensus.cc:2781] T 965bbe21dec14927b1823e97ad4fa5d1 P 9ed41cc926134f55b4e8216d60f3290a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:54:35.347646 18977 raft_consensus.cc:482] T 965bbe21dec14927b1823e97ad4fa5d1 P 9ed41cc926134f55b4e8216d60f3290a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:54:35.347682 18977 raft_consensus.cc:3037] T 965bbe21dec14927b1823e97ad4fa5d1 P 9ed41cc926134f55b4e8216d60f3290a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:54:35.348383 18977 raft_consensus.cc:504] T 965bbe21dec14927b1823e97ad4fa5d1 P 9ed41cc926134f55b4e8216d60f3290a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26697cf7e0624a09a7c2d456f27629d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40533 } } peers { permanent_uuid: "e4427d79675f4e0b91f3b6cd5d8358b5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46499 } } peers { permanent_uuid: "9ed41cc926134f55b4e8216d60f3290a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35175 } }
I20260926 20:54:35.348556 18977 leader_election.cc:288] T 965bbe21dec14927b1823e97ad4fa5d1 P 9ed41cc926134f55b4e8216d60f3290a [CANDIDATE]: Term 1 election: Requested vote from peers 26697cf7e0624a09a7c2d456f27629d0 (127.0.98.3:40533), e4427d79675f4e0b91f3b6cd5d8358b5 (127.0.98.2:46499)
I20260926 20:54:35.348703 18910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "965bbe21dec14927b1823e97ad4fa5d1" candidate_uuid: "9ed41cc926134f55b4e8216d60f3290a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26697cf7e0624a09a7c2d456f27629d0"
I20260926 20:54:35.348701 18779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "965bbe21dec14927b1823e97ad4fa5d1" candidate_uuid: "9ed41cc926134f55b4e8216d60f3290a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4427d79675f4e0b91f3b6cd5d8358b5"
I20260926 20:54:35.348776 18910 raft_consensus.cc:3037] T 965bbe21dec14927b1823e97ad4fa5d1 P 26697cf7e0624a09a7c2d456f27629d0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:54:35.348776 18779 raft_consensus.cc:3037] T 965bbe21dec14927b1823e97ad4fa5d1 P e4427d79675f4e0b91f3b6cd5d8358b5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:54:35.349641 18910 raft_consensus.cc:2454] T 965bbe21dec14927b1823e97ad4fa5d1 P 26697cf7e0624a09a7c2d456f27629d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ed41cc926134f55b4e8216d60f3290a in term 1.
I20260926 20:54:35.349641 18779 raft_consensus.cc:2454] T 965bbe21dec14927b1823e97ad4fa5d1 P e4427d79675f4e0b91f3b6cd5d8358b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ed41cc926134f55b4e8216d60f3290a in term 1.
I20260926 20:54:35.349798 18584 leader_election.cc:302] T 965bbe21dec14927b1823e97ad4fa5d1 P 9ed41cc926134f55b4e8216d60f3290a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ed41cc926134f55b4e8216d60f3290a, e4427d79675f4e0b91f3b6cd5d8358b5; no voters:
I20260926 20:54:35.349916 18977 raft_consensus.cc:2781] T 965bbe21dec14927b1823e97ad4fa5d1 P 9ed41cc926134f55b4e8216d60f3290a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:54:35.350057 18977 raft_consensus.cc:686] T 965bbe21dec14927b1823e97ad4fa5d1 P 9ed41cc926134f55b4e8216d60f3290a [term 1 LEADER]: Becoming Leader. State: Replica: 9ed41cc926134f55b4e8216d60f3290a, State: Running, Role: LEADER
I20260926 20:54:35.350148 18977 consensus_queue.cc:237] T 965bbe21dec14927b1823e97ad4fa5d1 P 9ed41cc926134f55b4e8216d60f3290a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26697cf7e0624a09a7c2d456f27629d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40533 } } peers { permanent_uuid: "e4427d79675f4e0b91f3b6cd5d8358b5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46499 } } peers { permanent_uuid: "9ed41cc926134f55b4e8216d60f3290a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35175 } }
I20260926 20:54:35.350953 18505 catalog_manager.cc:5495] T 965bbe21dec14927b1823e97ad4fa5d1 P 9ed41cc926134f55b4e8216d60f3290a reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ed41cc926134f55b4e8216d60f3290a (127.0.98.1). New cstate: current_term: 1 leader_uuid: "9ed41cc926134f55b4e8216d60f3290a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26697cf7e0624a09a7c2d456f27629d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4427d79675f4e0b91f3b6cd5d8358b5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ed41cc926134f55b4e8216d60f3290a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35175 } health_report { overall_health: HEALTHY } } }
I20260926 20:54:35.356396 18969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:35.354163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49914 (local address 127.0.98.1:35175)
0926 20:54:35.354205 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:35.354207 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:35.354216 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:54:35.354262 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:35.354263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:35.354286 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:35.354338 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:54:35.354421 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:35.354860 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:35.355693 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:35.355933 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:35.355990 (+ 57us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:35.356158 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:54:35.356238 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:54:35.356303 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:35.356322 (+ 19us) server_negotiation.cc:299] Negotiation successful
0926 20:54:35.356331 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 20:54:35.358253 18910 raft_consensus.cc:1261] T 965bbe21dec14927b1823e97ad4fa5d1 P 26697cf7e0624a09a7c2d456f27629d0 [term 1 FOLLOWER]: Refusing update from remote peer 9ed41cc926134f55b4e8216d60f3290a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:54:35.358480 18779 raft_consensus.cc:1261] T 965bbe21dec14927b1823e97ad4fa5d1 P e4427d79675f4e0b91f3b6cd5d8358b5 [term 1 FOLLOWER]: Refusing update from remote peer 9ed41cc926134f55b4e8216d60f3290a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:54:35.358582 18980 consensus_queue.cc:1035] T 965bbe21dec14927b1823e97ad4fa5d1 P 9ed41cc926134f55b4e8216d60f3290a [LEADER]: Connected to new peer: Peer: permanent_uuid: "26697cf7e0624a09a7c2d456f27629d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:54:35.358771 18980 consensus_queue.cc:1035] T 965bbe21dec14927b1823e97ad4fa5d1 P 9ed41cc926134f55b4e8216d60f3290a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4427d79675f4e0b91f3b6cd5d8358b5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:54:35.361171 18985 mvcc.cc:205] Tried to move back new op lower bound from 7333708084664365056 to 7333708084634726400. Current Snapshot: MvccSnapshot[applied={T|T < 7333708084664365056}]
I20260926 20:54:35.361166 18984 mvcc.cc:205] Tried to move back new op lower bound from 7333708084664365056 to 7333708084634726400. Current Snapshot: MvccSnapshot[applied={T|T < 7333708084664365056}]
I20260926 20:54:35.363605 18506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:54:35.363696 18506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:54:35.364405 18506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7955b388b4ac449e9f104453d92183c3: Sending DeleteTablet for 3 replicas of tablet 965bbe21dec14927b1823e97ad4fa5d1
I20260926 20:54:35.364712 18890 tablet_service.cc:1514] Processing DeleteTablet for tablet 965bbe21dec14927b1823e97ad4fa5d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:54:35 UTC) from {username='slave'} at 127.0.0.1:34030
I20260926 20:54:35.364751 18628 tablet_service.cc:1514] Processing DeleteTablet for tablet 965bbe21dec14927b1823e97ad4fa5d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:54:35 UTC) from {username='slave'} at 127.0.0.1:49904
I20260926 20:54:35.364768 18759 tablet_service.cc:1514] Processing DeleteTablet for tablet 965bbe21dec14927b1823e97ad4fa5d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:54:35 UTC) from {username='slave'} at 127.0.0.1:60944
I20260926 20:54:35.365072 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 18566
I20260926 20:54:35.365626 18993 tablet_replica.cc:330] T 965bbe21dec14927b1823e97ad4fa5d1 P 26697cf7e0624a09a7c2d456f27629d0: stopping tablet replica
I20260926 20:54:35.365746 18993 raft_consensus.cc:2229] T 965bbe21dec14927b1823e97ad4fa5d1 P 26697cf7e0624a09a7c2d456f27629d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:54:35.365866 18993 raft_consensus.cc:2258] T 965bbe21dec14927b1823e97ad4fa5d1 P 26697cf7e0624a09a7c2d456f27629d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:54:35.366035 18995 tablet_replica.cc:330] T 965bbe21dec14927b1823e97ad4fa5d1 P e4427d79675f4e0b91f3b6cd5d8358b5: stopping tablet replica
I20260926 20:54:35.366137 18995 raft_consensus.cc:2229] T 965bbe21dec14927b1823e97ad4fa5d1 P e4427d79675f4e0b91f3b6cd5d8358b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:54:35.366250 18995 raft_consensus.cc:2258] T 965bbe21dec14927b1823e97ad4fa5d1 P e4427d79675f4e0b91f3b6cd5d8358b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:54:35.366360 18993 ts_tablet_manager.cc:1905] T 965bbe21dec14927b1823e97ad4fa5d1 P 26697cf7e0624a09a7c2d456f27629d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:54:35.366740 18995 ts_tablet_manager.cc:1905] T 965bbe21dec14927b1823e97ad4fa5d1 P e4427d79675f4e0b91f3b6cd5d8358b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:54:35.367882 18993 ts_tablet_manager.cc:1918] T 965bbe21dec14927b1823e97ad4fa5d1 P 26697cf7e0624a09a7c2d456f27629d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:54:35.367967 18993 log.cc:1196] T 965bbe21dec14927b1823e97ad4fa5d1 P 26697cf7e0624a09a7c2d456f27629d0: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/ts-2/wal/wals/965bbe21dec14927b1823e97ad4fa5d1
I20260926 20:54:35.368285 18993 ts_tablet_manager.cc:1939] T 965bbe21dec14927b1823e97ad4fa5d1 P 26697cf7e0624a09a7c2d456f27629d0: Deleting consensus metadata
I20260926 20:54:35.368726 18491 catalog_manager.cc:4841] TS 26697cf7e0624a09a7c2d456f27629d0 (127.0.98.3:40533): tablet 965bbe21dec14927b1823e97ad4fa5d1 (table test-table [id=5b74c542b55445688ab01a7be388704d]) successfully deleted
I20260926 20:54:35.368817 18995 ts_tablet_manager.cc:1918] T 965bbe21dec14927b1823e97ad4fa5d1 P e4427d79675f4e0b91f3b6cd5d8358b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:54:35.368887 18995 log.cc:1196] T 965bbe21dec14927b1823e97ad4fa5d1 P e4427d79675f4e0b91f3b6cd5d8358b5: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455725523895-392-0/minicluster-data/ts-1/wal/wals/965bbe21dec14927b1823e97ad4fa5d1
I20260926 20:54:35.369155 18995 ts_tablet_manager.cc:1939] T 965bbe21dec14927b1823e97ad4fa5d1 P e4427d79675f4e0b91f3b6cd5d8358b5: Deleting consensus metadata
I20260926 20:54:35.369526 18493 catalog_manager.cc:4841] TS e4427d79675f4e0b91f3b6cd5d8358b5 (127.0.98.2:46499): tablet 965bbe21dec14927b1823e97ad4fa5d1 (table test-table [id=5b74c542b55445688ab01a7be388704d]) successfully deleted
W20260926 20:54:35.372377 18490 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35175 (error 108)
I20260926 20:54:35.372562 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 18697
I20260926 20:54:35.373332 18966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:35.372726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35175 (local address 127.0.0.1:49926)
0926 20:54:35.373268 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:35.373304 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":524}
W20260926 20:54:35.373462 18490 catalog_manager.cc:4584] TS 9ed41cc926134f55b4e8216d60f3290a (127.0.98.1:35175): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 965bbe21dec14927b1823e97ad4fa5d1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35175: connect: Connection refused (error 111)
I20260926 20:54:35.377444 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 18828
I20260926 20:54:35.382017 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 18475
2026-09-26T20:54:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (760 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:54:35 dist-test-slave-hwlj krb5kdc[18999](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:54:35 dist-test-slave-hwlj krb5kdc[18999](info): set up 2 sockets
Sep 26 20:54:35 dist-test-slave-hwlj krb5kdc[18999](info): commencing operation
krb5kdc: starting...
W20260926 20:54:36.427589 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/krb5kdc/test-user.keytab.
Sep 26 20:54:36 dist-test-slave-hwlj krb5kdc[18999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:54:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:54:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 20:54:36.547890 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40669
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40669
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:54:36.629288 19012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:54:36.629494 19012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:54:36.629518 19012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:54:36.630689 19012 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:54:36.630733 19012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:54:36.630746 19012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:54:36.630759 19012 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:54:36.630797 19012 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:54:36.632361 19012 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40669
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:54:36.632609 19012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:54:36.632838 19012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:54:36.637533 19020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:36.637540 19018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:36.637552 19017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:36.637662 19012 server_base.cc:1034] running on GCE node
I20260926 20:54:36.638022 19012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:54:36.638266 19012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:54:36.639411 19012 hybrid_clock.cc:648] HybridClock initialized: now 1790456076639401 us; error 31 us; skew 500 ppm
Sep 26 20:54:36 dist-test-slave-hwlj krb5kdc[18999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:54:36.641885 19012 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:54:36.642315 19012 webserver.cc:460] Webserver started at http://127.0.98.62:45455/ using document root <none> and password file <none>
I20260926 20:54:36.642549 19012 fs_manager.cc:359] Metadata directory not provided
I20260926 20:54:36.642613 19012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:54:36.642750 19012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:54:36.643852 19012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/master-0/data/instance:
uuid: "5cee713f2b8447b7b7f2596b861dffdd"
format_stamp: "Formatted at 2026-09-26 20:54:36 on dist-test-slave-hwlj"
server_key: "f67a7a9bee7bdf7b3329ebe324577571"
server_key_iv: "93a818deeaad1e77963da77745cbbc86"
server_key_version: "encryptionkey@0"
I20260926 20:54:36.644220 19012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/master-0/wal/instance:
uuid: "5cee713f2b8447b7b7f2596b861dffdd"
format_stamp: "Formatted at 2026-09-26 20:54:36 on dist-test-slave-hwlj"
server_key: "f67a7a9bee7bdf7b3329ebe324577571"
server_key_iv: "93a818deeaad1e77963da77745cbbc86"
server_key_version: "encryptionkey@0"
I20260926 20:54:36.645530 19012 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:54:36.646258 19027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:36.646430 19012 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:54:36.646545 19012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "5cee713f2b8447b7b7f2596b861dffdd"
format_stamp: "Formatted at 2026-09-26 20:54:36 on dist-test-slave-hwlj"
server_key: "f67a7a9bee7bdf7b3329ebe324577571"
server_key_iv: "93a818deeaad1e77963da77745cbbc86"
server_key_version: "encryptionkey@0"
I20260926 20:54:36.646615 19012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:54:36.657344 19012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:54:36.659077 19012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:54:36.659232 19012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:54:36.663156 19012 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40669
I20260926 20:54:36.663187 19079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40669 every 8 connection(s)
I20260926 20:54:36.663522 19012 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:54:36.664358 19080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:54:36.666561 19080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cee713f2b8447b7b7f2596b861dffdd: Bootstrap starting.
I20260926 20:54:36.667124 19080 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cee713f2b8447b7b7f2596b861dffdd: Neither blocks nor log segments found. Creating new log.
I20260926 20:54:36.667409 19080 log.cc:824] T 00000000000000000000000000000000 P 5cee713f2b8447b7b7f2596b861dffdd: Log is configured to *not* fsync() on all Append() calls
I20260926 20:54:36.668140 19080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cee713f2b8447b7b7f2596b861dffdd: No bootstrap required, opened a new log
I20260926 20:54:36.669715 19080 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cee713f2b8447b7b7f2596b861dffdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cee713f2b8447b7b7f2596b861dffdd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40669 } }
I20260926 20:54:36.669863 19080 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5cee713f2b8447b7b7f2596b861dffdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:54:36.669919 19080 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cee713f2b8447b7b7f2596b861dffdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cee713f2b8447b7b7f2596b861dffdd, State: Initialized, Role: FOLLOWER
I20260926 20:54:36.670063 19080 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cee713f2b8447b7b7f2596b861dffdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cee713f2b8447b7b7f2596b861dffdd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40669 } }
I20260926 20:54:36.670123 19080 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cee713f2b8447b7b7f2596b861dffdd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:54:36.670145 19080 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cee713f2b8447b7b7f2596b861dffdd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:54:36.670210 19080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cee713f2b8447b7b7f2596b861dffdd [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:54:36.670855 19080 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cee713f2b8447b7b7f2596b861dffdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cee713f2b8447b7b7f2596b861dffdd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40669 } }
I20260926 20:54:36.670966 19080 leader_election.cc:302] T 00000000000000000000000000000000 P 5cee713f2b8447b7b7f2596b861dffdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cee713f2b8447b7b7f2596b861dffdd; no voters:
I20260926 20:54:36.671147 19080 leader_election.cc:288] T 00000000000000000000000000000000 P 5cee713f2b8447b7b7f2596b861dffdd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:54:36.671180 19083 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cee713f2b8447b7b7f2596b861dffdd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:54:36.671322 19083 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cee713f2b8447b7b7f2596b861dffdd [term 1 LEADER]: Becoming Leader. State: Replica: 5cee713f2b8447b7b7f2596b861dffdd, State: Running, Role: LEADER
I20260926 20:54:36.671397 19080 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cee713f2b8447b7b7f2596b861dffdd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:54:36.671448 19083 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cee713f2b8447b7b7f2596b861dffdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cee713f2b8447b7b7f2596b861dffdd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40669 } }
I20260926 20:54:36.671926 19084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cee713f2b8447b7b7f2596b861dffdd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5cee713f2b8447b7b7f2596b861dffdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cee713f2b8447b7b7f2596b861dffdd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40669 } } }
I20260926 20:54:36.672009 19084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cee713f2b8447b7b7f2596b861dffdd [sys.catalog]: This master's current role is: LEADER
I20260926 20:54:36.672509 19085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cee713f2b8447b7b7f2596b861dffdd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cee713f2b8447b7b7f2596b861dffdd. Latest consensus state: current_term: 1 leader_uuid: "5cee713f2b8447b7b7f2596b861dffdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cee713f2b8447b7b7f2596b861dffdd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40669 } } }
I20260926 20:54:36.672600 19085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cee713f2b8447b7b7f2596b861dffdd [sys.catalog]: This master's current role is: LEADER
I20260926 20:54:36.672724 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 19012
I20260926 20:54:36.672807 19092 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:54:36.672857 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/master-0/wal/instance
I20260926 20:54:36.673070 392 external_mini_cluster.cc:1428] Setting key dc5050b1c451f5511903c1c90e7d5f5b
I20260926 20:54:36.673460 19092 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:54:36.676200 19092 catalog_manager.cc:1347] Generated new cluster ID: baa370845aaa4dd8b68eaba8a2714e55
I20260926 20:54:36.676255 19092 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 20:54:36 dist-test-slave-hwlj krb5kdc[18999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:54:36.684222 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:36.673748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46292 (local address 127.0.98.62:40669)
0926 20:54:36.673904 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:36.673913 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:36.673945 (+ 32us) server_negotiation.cc:407] Connection header received
0926 20:54:36.674054 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:36.674061 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:36.674119 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:36.674272 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:36.675080 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:36.676060 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:36.677102 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:36.677774 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:36.677898 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:36.680672 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:36.680677 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:36.680677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:54:36.680689 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:36.682583 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:36.682979 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:36.682981 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:36.682982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:36.683085 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:36.683287 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:36.683287 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:36.683288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:36.683571 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:36.683711 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:36.683864 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 20:54:36.684001 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":70,"threads_started":1}
I20260926 20:54:36.688905 19092 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:54:36.689451 19092 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:54:36.696470 19092 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5cee713f2b8447b7b7f2596b861dffdd: Generated new TSK 0
I20260926 20:54:36.696653 19092 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 20:54:36.724915 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40669
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:54:36.807653 19106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:54:36.807928 19106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:54:36.807968 19106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:54:36.809139 19106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:54:36.809203 19106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:54:36.809249 19106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:54:36.810788 19106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:54:36.811061 19106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:54:36.811286 19106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:54:36.816391 19112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:36.816432 19106 server_base.cc:1034] running on GCE node
W20260926 20:54:36.816391 19114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:36.816391 19111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:36.816797 19106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:54:36.817073 19106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:54:36.818233 19106 hybrid_clock.cc:648] HybridClock initialized: now 1790456076818211 us; error 37 us; skew 500 ppm
Sep 26 20:54:36 dist-test-slave-hwlj krb5kdc[18999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:54:36.820919 19106 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 20:54:36.821409 19106 webserver.cc:460] Webserver started at http://127.0.98.1:38237/ using document root <none> and password file <none>
I20260926 20:54:36.821641 19106 fs_manager.cc:359] Metadata directory not provided
I20260926 20:54:36.821705 19106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:54:36.821862 19106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:54:36.822922 19106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/ts-0/data/instance:
uuid: "92a76f2b2459464a9123863538dc009a"
format_stamp: "Formatted at 2026-09-26 20:54:36 on dist-test-slave-hwlj"
server_key: "b150cd6b71ba43ea6c8c103899b655dd"
server_key_iv: "b661250b11a8c9e3b1ec29215e2d381c"
server_key_version: "encryptionkey@0"
I20260926 20:54:36.823295 19106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/ts-0/wal/instance:
uuid: "92a76f2b2459464a9123863538dc009a"
format_stamp: "Formatted at 2026-09-26 20:54:36 on dist-test-slave-hwlj"
server_key: "b150cd6b71ba43ea6c8c103899b655dd"
server_key_iv: "b661250b11a8c9e3b1ec29215e2d381c"
server_key_version: "encryptionkey@0"
I20260926 20:54:36.824736 19106 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:54:36.825469 19121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:36.825629 19106 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:54:36.825723 19106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/ts-0/wal
uuid: "92a76f2b2459464a9123863538dc009a"
format_stamp: "Formatted at 2026-09-26 20:54:36 on dist-test-slave-hwlj"
server_key: "b150cd6b71ba43ea6c8c103899b655dd"
server_key_iv: "b661250b11a8c9e3b1ec29215e2d381c"
server_key_version: "encryptionkey@0"
I20260926 20:54:36.825814 19106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:54:36.840489 19106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:54:36.842300 19106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:54:36.842461 19106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:54:36.842690 19106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:54:36.842996 19106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:54:36.843046 19106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:36.843089 19106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:54:36.843117 19106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:36.849000 19106 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46841
I20260926 20:54:36.849047 19234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46841 every 8 connection(s)
I20260926 20:54:36.849380 19106 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:54:36.850068 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 19106
I20260926 20:54:36.850184 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/ts-0/wal/instance
I20260926 20:54:36.850428 392 external_mini_cluster.cc:1428] Setting key 9b7ae7415b9069c046a63a12b39c7ff7
Sep 26 20:54:36 dist-test-slave-hwlj krb5kdc[18999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 20:54:36.859843 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:36.850117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45789 (local address 127.0.98.62:40669)
0926 20:54:36.850169 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:36.850171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:36.850435 (+ 264us) server_negotiation.cc:407] Connection header received
0926 20:54:36.851654 (+ 1219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:36.851655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:36.851670 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:36.851727 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:36.852391 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:36.852908 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:36.853929 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:36.854220 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:36.854320 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:36.856696 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:36.856698 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:36.856698 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:54:36.856707 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:36.858451 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:36.858865 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:36.858866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:36.858867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:36.858928 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:36.859214 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:36.859215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:36.859215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:36.859335 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:36.859401 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:36.859641 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 20:54:36.859721 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 20:54:36.860067 19237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:36.850109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40669 (local address 127.0.98.1:45789)
0926 20:54:36.850319 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:36.850355 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:36.851548 (+ 1193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:36.851836 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:36.851845 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:36.852102 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:54:36.852297 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:36.852301 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:36.853003 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:36.853004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:36.853825 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:36.853827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:36.854327 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:36.854328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:36.854461 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:36.854985 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:54:36.854993 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:54:36.856598 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:54:36.858548 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:54:36.858556 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:54:36.858558 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:54:36.858787 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:54:36.859014 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:54:36.859014 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:54:36.859015 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:54:36.859126 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:54:36.859402 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:54:36.859408 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:54:36.859528 (+ 120us) client_negotiation.cc:769] Sending connection context
0926 20:54:36.859612 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 20:54:36.859792 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":111,"threads_started":1}
I20260926 20:54:36.860685 19235 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40669
I20260926 20:54:36.860824 19235 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 20:54:36.861034 19235 heartbeater.cc:511] Master 127.0.98.62:40669 requested a full tablet report, sending...
I20260926 20:54:36.861577 19044 ts_manager.cc:194] Registered new tserver with Master: 92a76f2b2459464a9123863538dc009a (127.0.98.1:46841)
I20260926 20:54:36.862288 19044 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45789
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 20:54:36.876418 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40669
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:54:36.957969 19240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:54:36.958215 19240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:54:36.958256 19240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:54:36.959453 19240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:54:36.959522 19240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:54:36.959570 19240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:54:36.961150 19240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:54:36.961459 19240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:54:36.961706 19240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:54:36.966861 19245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:36.966871 19246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:36.966881 19248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:36.966902 19240 server_base.cc:1034] running on GCE node
I20260926 20:54:36.967257 19240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:54:36.967471 19240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:54:36.968618 19240 hybrid_clock.cc:648] HybridClock initialized: now 1790456076968602 us; error 32 us; skew 500 ppm
Sep 26 20:54:36 dist-test-slave-hwlj krb5kdc[18999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:54:36.971186 19240 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 20:54:36.971685 19240 webserver.cc:460] Webserver started at http://127.0.98.2:42467/ using document root <none> and password file <none>
I20260926 20:54:36.971943 19240 fs_manager.cc:359] Metadata directory not provided
I20260926 20:54:36.971989 19240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:54:36.972128 19240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:54:36.973165 19240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/ts-1/data/instance:
uuid: "a72aae0e647942ea9a782940db238bda"
format_stamp: "Formatted at 2026-09-26 20:54:36 on dist-test-slave-hwlj"
server_key: "ca31a6dd52ec13fea11be0f4e0f3f5b0"
server_key_iv: "cddf1a4218d1d8a85185a8bfd4480b51"
server_key_version: "encryptionkey@0"
I20260926 20:54:36.973481 19240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a72aae0e647942ea9a782940db238bda"
format_stamp: "Formatted at 2026-09-26 20:54:36 on dist-test-slave-hwlj"
server_key: "ca31a6dd52ec13fea11be0f4e0f3f5b0"
server_key_iv: "cddf1a4218d1d8a85185a8bfd4480b51"
server_key_version: "encryptionkey@0"
I20260926 20:54:36.974838 19240 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:54:36.975490 19255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:36.975651 19240 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:54:36.975814 19240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/ts-1/wal
uuid: "a72aae0e647942ea9a782940db238bda"
format_stamp: "Formatted at 2026-09-26 20:54:36 on dist-test-slave-hwlj"
server_key: "ca31a6dd52ec13fea11be0f4e0f3f5b0"
server_key_iv: "cddf1a4218d1d8a85185a8bfd4480b51"
server_key_version: "encryptionkey@0"
I20260926 20:54:36.975906 19240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:54:37.007918 19240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:54:37.009795 19240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:54:37.009963 19240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:54:37.010200 19240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:54:37.010535 19240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:54:37.010587 19240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:37.010623 19240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:54:37.010650 19240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:37.016571 19240 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44415
I20260926 20:54:37.016618 19368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44415 every 8 connection(s)
I20260926 20:54:37.016940 19240 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
I20260926 20:54:37.021185 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 19240
I20260926 20:54:37.021306 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/ts-1/wal/instance
I20260926 20:54:37.021494 392 external_mini_cluster.cc:1428] Setting key e01b8cf778c639d48b31cadecad9df9a
Sep 26 20:54:37 dist-test-slave-hwlj krb5kdc[18999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:54:37.027921 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:37.017670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50877 (local address 127.0.98.62:40669)
0926 20:54:37.017729 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:37.017731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:37.017923 (+ 192us) server_negotiation.cc:407] Connection header received
0926 20:54:37.018850 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:37.018851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:37.018861 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:37.018918 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:37.019543 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:37.020132 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:37.021218 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:37.021542 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:37.021652 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:37.024474 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:37.024476 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:37.024476 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:54:37.024490 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:37.026192 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:37.026721 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:37.026722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:37.026722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:37.026774 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:37.027104 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:37.027106 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:37.027106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:37.027242 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:37.027307 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:37.027724 (+ 417us) server_negotiation.cc:299] Negotiation successful
0926 20:54:37.027854 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 20:54:37.028139 19371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:37.017642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40669 (local address 127.0.98.2:50877)
0926 20:54:37.017799 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:37.017840 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:37.018740 (+ 900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:37.018977 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:37.018981 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:37.019237 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:54:37.019434 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:37.019438 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:37.020241 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:37.020242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:37.021119 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:37.021121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:37.021719 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:37.021721 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:37.021850 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:37.022439 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:54:37.022452 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:54:37.024360 (+ 1908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:54:37.026279 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:54:37.026286 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:54:37.026289 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:54:37.026628 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:54:37.026842 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:54:37.026843 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:54:37.026844 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:54:37.027004 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:54:37.027318 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:54:37.027325 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:54:37.027513 (+ 188us) client_negotiation.cc:769] Sending connection context
0926 20:54:37.027646 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 20:54:37.027882 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":78,"threads_started":1}
I20260926 20:54:37.028836 19369 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40669
I20260926 20:54:37.028961 19369 heartbeater.cc:464] Registering TS with master...
I20260926 20:54:37.029160 19369 heartbeater.cc:511] Master 127.0.98.62:40669 requested a full tablet report, sending...
I20260926 20:54:37.029616 19044 ts_manager.cc:194] Registered new tserver with Master: a72aae0e647942ea9a782940db238bda (127.0.98.2:44415)
I20260926 20:54:37.029999 19044 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50877
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 20:54:37.048907 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40669
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:54:37.142284 19374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:54:37.142544 19374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:54:37.142583 19374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:54:37.143953 19374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:54:37.144011 19374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:54:37.144057 19374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:54:37.145565 19374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:54:37.145823 19374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:54:37.146054 19374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:54:37.151043 19382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:37.151119 19380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:37.151119 19379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:37.151199 19374 server_base.cc:1034] running on GCE node
I20260926 20:54:37.151566 19374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:54:37.151861 19374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:54:37.153014 19374 hybrid_clock.cc:648] HybridClock initialized: now 1790456077152996 us; error 33 us; skew 500 ppm
Sep 26 20:54:37 dist-test-slave-hwlj krb5kdc[18999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:54:37.155553 19374 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 20:54:37.156077 19374 webserver.cc:460] Webserver started at http://127.0.98.3:45701/ using document root <none> and password file <none>
I20260926 20:54:37.156312 19374 fs_manager.cc:359] Metadata directory not provided
I20260926 20:54:37.156376 19374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:54:37.156512 19374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:54:37.157604 19374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/ts-2/data/instance:
uuid: "2ad72a9dcbde4c09aea296e9e700252a"
format_stamp: "Formatted at 2026-09-26 20:54:37 on dist-test-slave-hwlj"
server_key: "6caf04e5068d1d929cc72f0ef22dd242"
server_key_iv: "1c964846b9c4c8d517c84419f805cfad"
server_key_version: "encryptionkey@0"
I20260926 20:54:37.157966 19374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2ad72a9dcbde4c09aea296e9e700252a"
format_stamp: "Formatted at 2026-09-26 20:54:37 on dist-test-slave-hwlj"
server_key: "6caf04e5068d1d929cc72f0ef22dd242"
server_key_iv: "1c964846b9c4c8d517c84419f805cfad"
server_key_version: "encryptionkey@0"
I20260926 20:54:37.159284 19374 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:54:37.160074 19389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:37.160246 19374 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:37.160326 19374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/ts-2/wal
uuid: "2ad72a9dcbde4c09aea296e9e700252a"
format_stamp: "Formatted at 2026-09-26 20:54:37 on dist-test-slave-hwlj"
server_key: "6caf04e5068d1d929cc72f0ef22dd242"
server_key_iv: "1c964846b9c4c8d517c84419f805cfad"
server_key_version: "encryptionkey@0"
I20260926 20:54:37.160377 19374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:54:37.175987 19374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:54:37.178188 19374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:54:37.178349 19374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:54:37.178610 19374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:54:37.178926 19374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:54:37.178961 19374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:37.178984 19374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:54:37.178997 19374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:37.184548 19374 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41835
I20260926 20:54:37.184589 19502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41835 every 8 connection(s)
I20260926 20:54:37.184921 19374 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 20:54:37 dist-test-slave-hwlj krb5kdc[18999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:54:37.193588 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:37.185573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49451 (local address 127.0.98.62:40669)
0926 20:54:37.185635 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:37.185637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:37.185849 (+ 212us) server_negotiation.cc:407] Connection header received
0926 20:54:37.186759 (+ 910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:37.186760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:37.186769 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:37.186820 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:37.187379 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:37.187929 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:37.188876 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:37.189052 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:37.189101 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:37.191127 (+ 2026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:37.191129 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:37.191129 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:54:37.191135 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:37.192496 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:37.192855 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:37.192856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:37.192856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:37.192903 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:37.193111 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:37.193112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:37.193112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:37.193204 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:37.193251 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:37.193466 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 20:54:37.193525 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 20:54:37.193852 19505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:37.185548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40669 (local address 127.0.98.3:49451)
0926 20:54:37.185724 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:37.185765 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:37.186657 (+ 892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:37.186878 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:37.186881 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:37.187126 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:54:37.187314 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:37.187318 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:37.188003 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:37.188004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:37.188806 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:37.188808 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:37.189150 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:37.189151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:37.189264 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:37.189720 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:54:37.189729 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:54:37.191041 (+ 1312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:54:37.192571 (+ 1530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:54:37.192577 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:54:37.192579 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:54:37.192790 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:54:37.192956 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:54:37.192957 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:54:37.192957 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:54:37.193061 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:54:37.193253 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:54:37.193257 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:54:37.193366 (+ 109us) client_negotiation.cc:769] Sending connection context
0926 20:54:37.193444 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 20:54:37.193555 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":82,"threads_started":1}
I20260926 20:54:37.193969 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 19374
I20260926 20:54:37.194046 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/ts-2/wal/instance
I20260926 20:54:37.194267 392 external_mini_cluster.cc:1428] Setting key 46852ecf2ca737b8b6ed0524d807f868
I20260926 20:54:37.194361 19503 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40669
I20260926 20:54:37.194478 19503 heartbeater.cc:464] Registering TS with master...
I20260926 20:54:37.194686 19503 heartbeater.cc:511] Master 127.0.98.62:40669 requested a full tablet report, sending...
I20260926 20:54:37.195080 19044 ts_manager.cc:194] Registered new tserver with Master: 2ad72a9dcbde4c09aea296e9e700252a (127.0.98.3:41835)
I20260926 20:54:37.195458 19044 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49451
I20260926 20:54:37.206054 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:54:37.213407 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:37.207471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46300 (local address 127.0.98.62:40669)
0926 20:54:37.207533 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:37.207535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:37.207653 (+ 118us) server_negotiation.cc:407] Connection header received
0926 20:54:37.207698 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:37.207699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:37.207708 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:37.207794 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:37.208331 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:37.208803 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:37.209485 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:37.209656 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:37.209702 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:37.210785 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:37.210786 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:37.210786 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:54:37.210792 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:37.212182 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:37.212497 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:37.212498 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:37.212498 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:37.212547 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:37.212755 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:37.212756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:37.212756 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:37.212976 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:37.213055 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:37.213221 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 20:54:37.213279 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 20:54:37.214938 19044 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46300:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:54:37.216615 19044 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:54:37.223079 19514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:37.219047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44415 (local address 127.0.0.1:50792)
0926 20:54:37.219375 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:37.219386 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:37.219456 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:37.219649 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:37.219653 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:37.219655 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:54:37.219729 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:37.219731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:37.220733 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:37.220734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:37.221879 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:37.221881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:37.222756 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:37.222757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:37.222835 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:37.222849 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 20:54:37.222946 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 20:54:37.223002 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":102,"threads_started":1}
I20260926 20:54:37.223088 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:37.219135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50792 (local address 127.0.98.2:44415)
0926 20:54:37.219327 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:37.219332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:37.219456 (+ 124us) server_negotiation.cc:407] Connection header received
0926 20:54:37.219530 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:37.219533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:37.219551 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:37.219625 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:54:37.219822 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:37.220623 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:37.221957 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:37.222675 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:37.222762 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:37.222827 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:37.222965 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 20:54:37.223007 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":74,"threads_started":1}
I20260926 20:54:37.223276 19515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:37.219042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41835 (local address 127.0.0.1:56608)
0926 20:54:37.219202 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:37.219246 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:37.219325 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:37.219797 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:37.219798 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:37.219801 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:54:37.219859 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:37.219861 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:37.221181 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:37.221182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:37.222161 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:37.222162 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:37.223032 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:37.223033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:37.223097 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:37.223101 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:54:37.223151 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 20:54:37.223189 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":34,"thread_start_us":60,"threads_started":1}
I20260926 20:54:37.223526 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:37.219216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56608 (local address 127.0.98.3:41835)
0926 20:54:37.219574 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:37.219579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:37.219591 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:54:37.219633 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:37.219636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:37.219652 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:37.219723 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:54:37.219992 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:37.221078 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:37.222246 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:37.222879 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:37.222946 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:37.222997 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:37.223382 (+ 385us) server_negotiation.cc:299] Negotiation successful
0926 20:54:37.223427 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":80,"threads_started":1}
I20260926 20:54:37.223627 19518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:37.219250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46841 (local address 127.0.0.1:35876)
0926 20:54:37.219400 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:37.219410 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:37.219456 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:37.220085 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:37.220086 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:37.220088 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:54:37.220136 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:37.220137 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:37.221322 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:37.221323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:37.222437 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:37.222440 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:37.223394 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:37.223395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:37.223457 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:37.223462 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:54:37.223511 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 20:54:37.223552 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":67,"threads_started":1}
I20260926 20:54:37.223938 19519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:37.219455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35876 (local address 127.0.98.1:46841)
0926 20:54:37.219611 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:37.219614 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:37.219623 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:54:37.219658 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:37.219660 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:37.219678 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:37.220082 (+ 404us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:54:37.220199 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:37.221217 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:37.222520 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:37.223234 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:37.223293 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:37.223361 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:37.223806 (+ 445us) server_negotiation.cc:299] Negotiation successful
0926 20:54:37.223853 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":78,"threads_started":1}
I20260926 20:54:37.224002 19303 tablet_service.cc:1467] Processing CreateTablet for tablet 55d73195f0814335922273a6147f8bc5 (DEFAULT_TABLE table=test-table [id=0e7bed2a60834affb7d3ff416183fd8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:54:37.224238 19437 tablet_service.cc:1467] Processing CreateTablet for tablet 55d73195f0814335922273a6147f8bc5 (DEFAULT_TABLE table=test-table [id=0e7bed2a60834affb7d3ff416183fd8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:54:37.224371 19303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55d73195f0814335922273a6147f8bc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:54:37.224553 19437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55d73195f0814335922273a6147f8bc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:54:37.224527 19169 tablet_service.cc:1467] Processing CreateTablet for tablet 55d73195f0814335922273a6147f8bc5 (DEFAULT_TABLE table=test-table [id=0e7bed2a60834affb7d3ff416183fd8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:54:37.224855 19169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55d73195f0814335922273a6147f8bc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:54:37.227195 19520 tablet_bootstrap.cc:492] T 55d73195f0814335922273a6147f8bc5 P 92a76f2b2459464a9123863538dc009a: Bootstrap starting.
I20260926 20:54:37.227635 19522 tablet_bootstrap.cc:492] T 55d73195f0814335922273a6147f8bc5 P a72aae0e647942ea9a782940db238bda: Bootstrap starting.
I20260926 20:54:37.228024 19520 tablet_bootstrap.cc:654] T 55d73195f0814335922273a6147f8bc5 P 92a76f2b2459464a9123863538dc009a: Neither blocks nor log segments found. Creating new log.
I20260926 20:54:37.228284 19522 tablet_bootstrap.cc:654] T 55d73195f0814335922273a6147f8bc5 P a72aae0e647942ea9a782940db238bda: Neither blocks nor log segments found. Creating new log.
I20260926 20:54:37.228693 19520 log.cc:824] T 55d73195f0814335922273a6147f8bc5 P 92a76f2b2459464a9123863538dc009a: Log is configured to *not* fsync() on all Append() calls
I20260926 20:54:37.228910 19522 log.cc:824] T 55d73195f0814335922273a6147f8bc5 P a72aae0e647942ea9a782940db238bda: Log is configured to *not* fsync() on all Append() calls
I20260926 20:54:37.229270 19521 tablet_bootstrap.cc:492] T 55d73195f0814335922273a6147f8bc5 P 2ad72a9dcbde4c09aea296e9e700252a: Bootstrap starting.
I20260926 20:54:37.229674 19522 tablet_bootstrap.cc:492] T 55d73195f0814335922273a6147f8bc5 P a72aae0e647942ea9a782940db238bda: No bootstrap required, opened a new log
I20260926 20:54:37.229687 19520 tablet_bootstrap.cc:492] T 55d73195f0814335922273a6147f8bc5 P 92a76f2b2459464a9123863538dc009a: No bootstrap required, opened a new log
I20260926 20:54:37.229750 19520 ts_tablet_manager.cc:1397] T 55d73195f0814335922273a6147f8bc5 P 92a76f2b2459464a9123863538dc009a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 20:54:37.229797 19522 ts_tablet_manager.cc:1397] T 55d73195f0814335922273a6147f8bc5 P a72aae0e647942ea9a782940db238bda: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:54:37.229883 19521 tablet_bootstrap.cc:654] T 55d73195f0814335922273a6147f8bc5 P 2ad72a9dcbde4c09aea296e9e700252a: Neither blocks nor log segments found. Creating new log.
I20260926 20:54:37.230341 19521 log.cc:824] T 55d73195f0814335922273a6147f8bc5 P 2ad72a9dcbde4c09aea296e9e700252a: Log is configured to *not* fsync() on all Append() calls
I20260926 20:54:37.231130 19521 tablet_bootstrap.cc:492] T 55d73195f0814335922273a6147f8bc5 P 2ad72a9dcbde4c09aea296e9e700252a: No bootstrap required, opened a new log
I20260926 20:54:37.231211 19521 ts_tablet_manager.cc:1397] T 55d73195f0814335922273a6147f8bc5 P 2ad72a9dcbde4c09aea296e9e700252a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:54:37.231830 19522 raft_consensus.cc:348] T 55d73195f0814335922273a6147f8bc5 P a72aae0e647942ea9a782940db238bda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ad72a9dcbde4c09aea296e9e700252a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41835 } } peers { permanent_uuid: "a72aae0e647942ea9a782940db238bda" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44415 } } peers { permanent_uuid: "92a76f2b2459464a9123863538dc009a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46841 } }
I20260926 20:54:37.231830 19520 raft_consensus.cc:348] T 55d73195f0814335922273a6147f8bc5 P 92a76f2b2459464a9123863538dc009a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ad72a9dcbde4c09aea296e9e700252a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41835 } } peers { permanent_uuid: "a72aae0e647942ea9a782940db238bda" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44415 } } peers { permanent_uuid: "92a76f2b2459464a9123863538dc009a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46841 } }
I20260926 20:54:37.232023 19520 raft_consensus.cc:374] T 55d73195f0814335922273a6147f8bc5 P 92a76f2b2459464a9123863538dc009a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:54:37.232023 19522 raft_consensus.cc:374] T 55d73195f0814335922273a6147f8bc5 P a72aae0e647942ea9a782940db238bda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:54:37.232060 19522 raft_consensus.cc:729] T 55d73195f0814335922273a6147f8bc5 P a72aae0e647942ea9a782940db238bda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a72aae0e647942ea9a782940db238bda, State: Initialized, Role: FOLLOWER
I20260926 20:54:37.232060 19520 raft_consensus.cc:729] T 55d73195f0814335922273a6147f8bc5 P 92a76f2b2459464a9123863538dc009a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92a76f2b2459464a9123863538dc009a, State: Initialized, Role: FOLLOWER
I20260926 20:54:37.232205 19520 consensus_queue.cc:260] T 55d73195f0814335922273a6147f8bc5 P 92a76f2b2459464a9123863538dc009a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ad72a9dcbde4c09aea296e9e700252a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41835 } } peers { permanent_uuid: "a72aae0e647942ea9a782940db238bda" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44415 } } peers { permanent_uuid: "92a76f2b2459464a9123863538dc009a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46841 } }
I20260926 20:54:37.232220 19522 consensus_queue.cc:260] T 55d73195f0814335922273a6147f8bc5 P a72aae0e647942ea9a782940db238bda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ad72a9dcbde4c09aea296e9e700252a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41835 } } peers { permanent_uuid: "a72aae0e647942ea9a782940db238bda" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44415 } } peers { permanent_uuid: "92a76f2b2459464a9123863538dc009a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46841 } }
I20260926 20:54:37.232502 19520 ts_tablet_manager.cc:1428] T 55d73195f0814335922273a6147f8bc5 P 92a76f2b2459464a9123863538dc009a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:54:37.232502 19522 ts_tablet_manager.cc:1428] T 55d73195f0814335922273a6147f8bc5 P a72aae0e647942ea9a782940db238bda: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:54:37.232625 19369 heartbeater.cc:503] Master 127.0.98.62:40669 was elected leader, sending a full tablet report...
I20260926 20:54:37.232635 19235 heartbeater.cc:503] Master 127.0.98.62:40669 was elected leader, sending a full tablet report...
I20260926 20:54:37.233037 19521 raft_consensus.cc:348] T 55d73195f0814335922273a6147f8bc5 P 2ad72a9dcbde4c09aea296e9e700252a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ad72a9dcbde4c09aea296e9e700252a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41835 } } peers { permanent_uuid: "a72aae0e647942ea9a782940db238bda" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44415 } } peers { permanent_uuid: "92a76f2b2459464a9123863538dc009a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46841 } }
I20260926 20:54:37.233210 19521 raft_consensus.cc:374] T 55d73195f0814335922273a6147f8bc5 P 2ad72a9dcbde4c09aea296e9e700252a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:54:37.233244 19521 raft_consensus.cc:729] T 55d73195f0814335922273a6147f8bc5 P 2ad72a9dcbde4c09aea296e9e700252a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ad72a9dcbde4c09aea296e9e700252a, State: Initialized, Role: FOLLOWER
I20260926 20:54:37.233372 19521 consensus_queue.cc:260] T 55d73195f0814335922273a6147f8bc5 P 2ad72a9dcbde4c09aea296e9e700252a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ad72a9dcbde4c09aea296e9e700252a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41835 } } peers { permanent_uuid: "a72aae0e647942ea9a782940db238bda" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44415 } } peers { permanent_uuid: "92a76f2b2459464a9123863538dc009a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46841 } }
I20260926 20:54:37.233721 19521 ts_tablet_manager.cc:1428] T 55d73195f0814335922273a6147f8bc5 P 2ad72a9dcbde4c09aea296e9e700252a: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260926 20:54:37.233855 19503 heartbeater.cc:503] Master 127.0.98.62:40669 was elected leader, sending a full tablet report...
W20260926 20:54:37.267506 19370 tablet.cc:2369] T 55d73195f0814335922273a6147f8bc5 P a72aae0e647942ea9a782940db238bda: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:54:37.350059 19236 tablet.cc:2369] T 55d73195f0814335922273a6147f8bc5 P 92a76f2b2459464a9123863538dc009a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:54:37.392633 19528 raft_consensus.cc:482] T 55d73195f0814335922273a6147f8bc5 P 2ad72a9dcbde4c09aea296e9e700252a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:54:37.392813 19528 raft_consensus.cc:504] T 55d73195f0814335922273a6147f8bc5 P 2ad72a9dcbde4c09aea296e9e700252a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ad72a9dcbde4c09aea296e9e700252a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41835 } } peers { permanent_uuid: "a72aae0e647942ea9a782940db238bda" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44415 } } peers { permanent_uuid: "92a76f2b2459464a9123863538dc009a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46841 } }
I20260926 20:54:37.393190 19528 leader_election.cc:288] T 55d73195f0814335922273a6147f8bc5 P 2ad72a9dcbde4c09aea296e9e700252a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a72aae0e647942ea9a782940db238bda (127.0.98.2:44415), 92a76f2b2459464a9123863538dc009a (127.0.98.1:46841)
I20260926 20:54:37.396093 19505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:37.393401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44415 (local address 127.0.98.3:42809)
0926 20:54:37.393456 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:37.393469 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:37.393516 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:37.393782 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:37.393783 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:37.393785 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:54:37.393858 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:37.393860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:37.394495 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:37.394496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:37.395243 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:37.395245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:37.395837 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:37.395838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:37.395916 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:37.395919 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:54:37.395984 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 20:54:37.396022 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 20:54:37.396093 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:37.393609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42809 (local address 127.0.98.2:44415)
0926 20:54:37.393657 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:37.393659 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:37.393667 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:54:37.393697 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:37.393698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:37.393711 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:37.393766 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:54:37.393929 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:37.394412 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:37.395336 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:37.395709 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:37.395834 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:37.395845 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:37.395990 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 20:54:37.396027 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 20:54:37.396373 19529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:37.393532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46841 (local address 127.0.98.3:53837)
0926 20:54:37.393700 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:37.393715 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:37.393818 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:37.393980 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:37.393981 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:37.393982 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:54:37.394070 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:37.394071 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:37.394679 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:37.394680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:37.395415 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:37.395417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:37.396065 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:37.396066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:37.396141 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:37.396160 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 20:54:37.396251 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 20:54:37.396301 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":112,"threads_started":1}
I20260926 20:54:37.396394 19519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:37.393660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53837 (local address 127.0.98.1:46841)
0926 20:54:37.393712 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:37.393714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:37.393785 (+ 71us) server_negotiation.cc:407] Connection header received
0926 20:54:37.393897 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:37.393898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:37.393909 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:37.393977 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:54:37.394138 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:37.394596 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:37.395502 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:37.395985 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:37.396063 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:37.396072 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:37.396292 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 20:54:37.396334 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 20:54:37.396430 19323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d73195f0814335922273a6147f8bc5" candidate_uuid: "2ad72a9dcbde4c09aea296e9e700252a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a72aae0e647942ea9a782940db238bda" is_pre_election: true
I20260926 20:54:37.396621 19323 raft_consensus.cc:2454] T 55d73195f0814335922273a6147f8bc5 P a72aae0e647942ea9a782940db238bda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ad72a9dcbde4c09aea296e9e700252a in term 0.
I20260926 20:54:37.396831 19189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d73195f0814335922273a6147f8bc5" candidate_uuid: "2ad72a9dcbde4c09aea296e9e700252a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92a76f2b2459464a9123863538dc009a" is_pre_election: true
I20260926 20:54:37.396885 19390 leader_election.cc:302] T 55d73195f0814335922273a6147f8bc5 P 2ad72a9dcbde4c09aea296e9e700252a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ad72a9dcbde4c09aea296e9e700252a, a72aae0e647942ea9a782940db238bda; no voters:
I20260926 20:54:37.396972 19189 raft_consensus.cc:2454] T 55d73195f0814335922273a6147f8bc5 P 92a76f2b2459464a9123863538dc009a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ad72a9dcbde4c09aea296e9e700252a in term 0.
I20260926 20:54:37.397003 19528 raft_consensus.cc:2781] T 55d73195f0814335922273a6147f8bc5 P 2ad72a9dcbde4c09aea296e9e700252a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:54:37.397071 19528 raft_consensus.cc:482] T 55d73195f0814335922273a6147f8bc5 P 2ad72a9dcbde4c09aea296e9e700252a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:54:37.397097 19528 raft_consensus.cc:3037] T 55d73195f0814335922273a6147f8bc5 P 2ad72a9dcbde4c09aea296e9e700252a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:54:37.397804 19528 raft_consensus.cc:504] T 55d73195f0814335922273a6147f8bc5 P 2ad72a9dcbde4c09aea296e9e700252a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ad72a9dcbde4c09aea296e9e700252a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41835 } } peers { permanent_uuid: "a72aae0e647942ea9a782940db238bda" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44415 } } peers { permanent_uuid: "92a76f2b2459464a9123863538dc009a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46841 } }
I20260926 20:54:37.397935 19528 leader_election.cc:288] T 55d73195f0814335922273a6147f8bc5 P 2ad72a9dcbde4c09aea296e9e700252a [CANDIDATE]: Term 1 election: Requested vote from peers a72aae0e647942ea9a782940db238bda (127.0.98.2:44415), 92a76f2b2459464a9123863538dc009a (127.0.98.1:46841)
I20260926 20:54:37.398099 19323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d73195f0814335922273a6147f8bc5" candidate_uuid: "2ad72a9dcbde4c09aea296e9e700252a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a72aae0e647942ea9a782940db238bda"
I20260926 20:54:37.398138 19189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d73195f0814335922273a6147f8bc5" candidate_uuid: "2ad72a9dcbde4c09aea296e9e700252a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92a76f2b2459464a9123863538dc009a"
I20260926 20:54:37.398198 19323 raft_consensus.cc:3037] T 55d73195f0814335922273a6147f8bc5 P a72aae0e647942ea9a782940db238bda [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:54:37.398197 19189 raft_consensus.cc:3037] T 55d73195f0814335922273a6147f8bc5 P 92a76f2b2459464a9123863538dc009a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:54:37.398893 19323 raft_consensus.cc:2454] T 55d73195f0814335922273a6147f8bc5 P a72aae0e647942ea9a782940db238bda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ad72a9dcbde4c09aea296e9e700252a in term 1.
I20260926 20:54:37.398893 19189 raft_consensus.cc:2454] T 55d73195f0814335922273a6147f8bc5 P 92a76f2b2459464a9123863538dc009a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ad72a9dcbde4c09aea296e9e700252a in term 1.
I20260926 20:54:37.399065 19390 leader_election.cc:302] T 55d73195f0814335922273a6147f8bc5 P 2ad72a9dcbde4c09aea296e9e700252a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ad72a9dcbde4c09aea296e9e700252a, a72aae0e647942ea9a782940db238bda; no voters:
I20260926 20:54:37.399174 19528 raft_consensus.cc:2781] T 55d73195f0814335922273a6147f8bc5 P 2ad72a9dcbde4c09aea296e9e700252a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:54:37.399334 19528 raft_consensus.cc:686] T 55d73195f0814335922273a6147f8bc5 P 2ad72a9dcbde4c09aea296e9e700252a [term 1 LEADER]: Becoming Leader. State: Replica: 2ad72a9dcbde4c09aea296e9e700252a, State: Running, Role: LEADER
I20260926 20:54:37.399425 19528 consensus_queue.cc:237] T 55d73195f0814335922273a6147f8bc5 P 2ad72a9dcbde4c09aea296e9e700252a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ad72a9dcbde4c09aea296e9e700252a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41835 } } peers { permanent_uuid: "a72aae0e647942ea9a782940db238bda" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44415 } } peers { permanent_uuid: "92a76f2b2459464a9123863538dc009a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46841 } }
I20260926 20:54:37.400311 19043 catalog_manager.cc:5495] T 55d73195f0814335922273a6147f8bc5 P 2ad72a9dcbde4c09aea296e9e700252a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ad72a9dcbde4c09aea296e9e700252a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "2ad72a9dcbde4c09aea296e9e700252a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ad72a9dcbde4c09aea296e9e700252a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a72aae0e647942ea9a782940db238bda" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92a76f2b2459464a9123863538dc009a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46841 } health_report { overall_health: UNKNOWN } } }
I20260926 20:54:37.407450 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:37.405183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56618 (local address 127.0.98.3:41835)
0926 20:54:37.405229 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:37.405230 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:37.405238 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:54:37.405311 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:37.405312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:37.405326 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:37.405381 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:54:37.405492 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:37.405937 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:37.406705 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:37.406908 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:37.406965 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:37.407099 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:54:37.407197 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:54:37.407255 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:37.407330 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 20:54:37.407354 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 20:54:37.409161 19189 raft_consensus.cc:1261] T 55d73195f0814335922273a6147f8bc5 P 92a76f2b2459464a9123863538dc009a [term 1 FOLLOWER]: Refusing update from remote peer 2ad72a9dcbde4c09aea296e9e700252a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:54:37.409157 19323 raft_consensus.cc:1261] T 55d73195f0814335922273a6147f8bc5 P a72aae0e647942ea9a782940db238bda [term 1 FOLLOWER]: Refusing update from remote peer 2ad72a9dcbde4c09aea296e9e700252a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:54:37.409482 19530 consensus_queue.cc:1035] T 55d73195f0814335922273a6147f8bc5 P 2ad72a9dcbde4c09aea296e9e700252a [LEADER]: Connected to new peer: Peer: permanent_uuid: "92a76f2b2459464a9123863538dc009a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:54:37.409608 19528 consensus_queue.cc:1035] T 55d73195f0814335922273a6147f8bc5 P 2ad72a9dcbde4c09aea296e9e700252a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a72aae0e647942ea9a782940db238bda" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:54:37.411717 19533 mvcc.cc:205] Tried to move back new op lower bound from 7333708093065342976 to 7333708093028732928. Current Snapshot: MvccSnapshot[applied={T|T < 7333708093065342976}]
I20260926 20:54:37.412514 19535 mvcc.cc:205] Tried to move back new op lower bound from 7333708093065342976 to 7333708093028732928. Current Snapshot: MvccSnapshot[applied={T|T < 7333708093065342976}]
I20260926 20:54:37.413348 19534 mvcc.cc:205] Tried to move back new op lower bound from 7333708093065342976 to 7333708093028732928. Current Snapshot: MvccSnapshot[applied={T|T < 7333708093065342976}]
I20260926 20:54:37.414186 19044 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46300:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:54:37.414286 19044 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46300:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:54:37.415005 19044 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5cee713f2b8447b7b7f2596b861dffdd: Sending DeleteTablet for 3 replicas of tablet 55d73195f0814335922273a6147f8bc5
I20260926 20:54:37.415292 19437 tablet_service.cc:1514] Processing DeleteTablet for tablet 55d73195f0814335922273a6147f8bc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:54:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56608
I20260926 20:54:37.415354 19169 tablet_service.cc:1514] Processing DeleteTablet for tablet 55d73195f0814335922273a6147f8bc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:54:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35876
I20260926 20:54:37.415371 19303 tablet_service.cc:1514] Processing DeleteTablet for tablet 55d73195f0814335922273a6147f8bc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:54:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50792
I20260926 20:54:37.415512 19542 tablet_replica.cc:330] T 55d73195f0814335922273a6147f8bc5 P 2ad72a9dcbde4c09aea296e9e700252a: stopping tablet replica
I20260926 20:54:37.415585 19543 tablet_replica.cc:330] T 55d73195f0814335922273a6147f8bc5 P 92a76f2b2459464a9123863538dc009a: stopping tablet replica
I20260926 20:54:37.415593 19542 raft_consensus.cc:2229] T 55d73195f0814335922273a6147f8bc5 P 2ad72a9dcbde4c09aea296e9e700252a [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:54:37.415675 19542 raft_consensus.cc:2258] T 55d73195f0814335922273a6147f8bc5 P 2ad72a9dcbde4c09aea296e9e700252a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:54:37.415675 19543 raft_consensus.cc:2229] T 55d73195f0814335922273a6147f8bc5 P 92a76f2b2459464a9123863538dc009a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:54:37.415792 19543 raft_consensus.cc:2258] T 55d73195f0814335922273a6147f8bc5 P 92a76f2b2459464a9123863538dc009a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:54:37.415902 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 19106
I20260926 20:54:37.416126 19542 ts_tablet_manager.cc:1905] T 55d73195f0814335922273a6147f8bc5 P 2ad72a9dcbde4c09aea296e9e700252a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:54:37.418309 19542 ts_tablet_manager.cc:1918] T 55d73195f0814335922273a6147f8bc5 P 2ad72a9dcbde4c09aea296e9e700252a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:54:37.418390 19542 log.cc:1196] T 55d73195f0814335922273a6147f8bc5 P 2ad72a9dcbde4c09aea296e9e700252a: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455725523895-392-0/minicluster-data/ts-2/wal/wals/55d73195f0814335922273a6147f8bc5
I20260926 20:54:37.418669 19542 ts_tablet_manager.cc:1939] T 55d73195f0814335922273a6147f8bc5 P 2ad72a9dcbde4c09aea296e9e700252a: Deleting consensus metadata
I20260926 20:54:37.419068 19030 catalog_manager.cc:4841] TS 2ad72a9dcbde4c09aea296e9e700252a (127.0.98.3:41835): tablet 55d73195f0814335922273a6147f8bc5 (table test-table [id=0e7bed2a60834affb7d3ff416183fd8a]) successfully deleted
W20260926 20:54:37.421937 19028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46841 (error 108)
I20260926 20:54:37.422185 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 19240
I20260926 20:54:37.422422 19518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:37.422304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46841 (local address 127.0.0.1:35888)
0926 20:54:37.422351 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:37.422386 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260926 20:54:37.422525 19028 catalog_manager.cc:4584] TS 92a76f2b2459464a9123863538dc009a (127.0.98.1:46841): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 55d73195f0814335922273a6147f8bc5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46841: connect: Connection refused (error 111)
I20260926 20:54:37.427498 19518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:37.427392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44415 (local address 127.0.0.1:50798)
0926 20:54:37.427440 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:37.427467 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 20:54:37.427683 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 19374
I20260926 20:54:37.432564 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 19012
2026-09-26T20:54:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2050 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T20:54:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:54:37Z Disabled control of system clock
I20260926 20:54:37.459301 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35699
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 20:54:37.540560 19549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:54:37.540791 19549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:54:37.540810 19549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:54:37.541900 19549 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:54:37.541937 19549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:54:37.541949 19549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:54:37.541962 19549 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:54:37.541975 19549 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:54:37.543340 19549 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35699
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:54:37.543541 19549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:54:37.543813 19549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:54:37.549176 19557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:37.549216 19555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:37.549360 19554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:37.550112 19549 server_base.cc:1034] running on GCE node
I20260926 20:54:37.550369 19549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:54:37.550657 19549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:54:37.551819 19549 hybrid_clock.cc:648] HybridClock initialized: now 1790456077551801 us; error 31 us; skew 500 ppm
I20260926 20:54:37.553113 19549 webserver.cc:460] Webserver started at http://127.0.98.62:42513/ using document root <none> and password file <none>
I20260926 20:54:37.553372 19549 fs_manager.cc:359] Metadata directory not provided
I20260926 20:54:37.553442 19549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:54:37.553582 19549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:54:37.554670 19549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/master-0/data/instance:
uuid: "bae0d7d585f345e5a84d1ab39861d9ec"
format_stamp: "Formatted at 2026-09-26 20:54:37 on dist-test-slave-hwlj"
server_key: "35363fd83ee442c723bcb61a2c7b2dc5"
server_key_iv: "5e27d875dcaa7bd6a06948ed8cbc359c"
server_key_version: "encryptionkey@0"
I20260926 20:54:37.555028 19549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/master-0/wal/instance:
uuid: "bae0d7d585f345e5a84d1ab39861d9ec"
format_stamp: "Formatted at 2026-09-26 20:54:37 on dist-test-slave-hwlj"
server_key: "35363fd83ee442c723bcb61a2c7b2dc5"
server_key_iv: "5e27d875dcaa7bd6a06948ed8cbc359c"
server_key_version: "encryptionkey@0"
I20260926 20:54:37.556443 19549 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:54:37.557164 19563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:37.557330 19549 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:54:37.557430 19549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "bae0d7d585f345e5a84d1ab39861d9ec"
format_stamp: "Formatted at 2026-09-26 20:54:37 on dist-test-slave-hwlj"
server_key: "35363fd83ee442c723bcb61a2c7b2dc5"
server_key_iv: "5e27d875dcaa7bd6a06948ed8cbc359c"
server_key_version: "encryptionkey@0"
I20260926 20:54:37.557502 19549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:54:37.574790 19549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:54:37.575165 19549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:54:37.575323 19549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:54:37.579424 19549 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35699
I20260926 20:54:37.579461 19615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35699 every 8 connection(s)
I20260926 20:54:37.579854 19549 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:54:37.580852 19616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:54:37.583076 19616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bae0d7d585f345e5a84d1ab39861d9ec: Bootstrap starting.
I20260926 20:54:37.583547 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 19549
I20260926 20:54:37.583671 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/master-0/wal/instance
I20260926 20:54:37.583666 19616 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bae0d7d585f345e5a84d1ab39861d9ec: Neither blocks nor log segments found. Creating new log.
I20260926 20:54:37.583910 392 external_mini_cluster.cc:1428] Setting key 1f1c15f214ce68ed09969c30065107ef
I20260926 20:54:37.584041 19616 log.cc:824] T 00000000000000000000000000000000 P bae0d7d585f345e5a84d1ab39861d9ec: Log is configured to *not* fsync() on all Append() calls
I20260926 20:54:37.584884 19616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bae0d7d585f345e5a84d1ab39861d9ec: No bootstrap required, opened a new log
I20260926 20:54:37.587477 19616 raft_consensus.cc:348] T 00000000000000000000000000000000 P bae0d7d585f345e5a84d1ab39861d9ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae0d7d585f345e5a84d1ab39861d9ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35699 } }
I20260926 20:54:37.587639 19616 raft_consensus.cc:374] T 00000000000000000000000000000000 P bae0d7d585f345e5a84d1ab39861d9ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:54:37.587685 19616 raft_consensus.cc:729] T 00000000000000000000000000000000 P bae0d7d585f345e5a84d1ab39861d9ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bae0d7d585f345e5a84d1ab39861d9ec, State: Initialized, Role: FOLLOWER
I20260926 20:54:37.587849 19616 consensus_queue.cc:260] T 00000000000000000000000000000000 P bae0d7d585f345e5a84d1ab39861d9ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae0d7d585f345e5a84d1ab39861d9ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35699 } }
I20260926 20:54:37.587949 19616 raft_consensus.cc:388] T 00000000000000000000000000000000 P bae0d7d585f345e5a84d1ab39861d9ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:54:37.587992 19616 raft_consensus.cc:482] T 00000000000000000000000000000000 P bae0d7d585f345e5a84d1ab39861d9ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:54:37.588042 19616 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bae0d7d585f345e5a84d1ab39861d9ec [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:54:37.588752 19616 raft_consensus.cc:504] T 00000000000000000000000000000000 P bae0d7d585f345e5a84d1ab39861d9ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae0d7d585f345e5a84d1ab39861d9ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35699 } }
I20260926 20:54:37.588874 19616 leader_election.cc:302] T 00000000000000000000000000000000 P bae0d7d585f345e5a84d1ab39861d9ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bae0d7d585f345e5a84d1ab39861d9ec; no voters:
I20260926 20:54:37.589052 19616 leader_election.cc:288] T 00000000000000000000000000000000 P bae0d7d585f345e5a84d1ab39861d9ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:54:37.589143 19621 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bae0d7d585f345e5a84d1ab39861d9ec [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:54:37.589301 19621 raft_consensus.cc:686] T 00000000000000000000000000000000 P bae0d7d585f345e5a84d1ab39861d9ec [term 1 LEADER]: Becoming Leader. State: Replica: bae0d7d585f345e5a84d1ab39861d9ec, State: Running, Role: LEADER
I20260926 20:54:37.589375 19616 sys_catalog.cc:564] T 00000000000000000000000000000000 P bae0d7d585f345e5a84d1ab39861d9ec [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:54:37.589466 19621 consensus_queue.cc:237] T 00000000000000000000000000000000 P bae0d7d585f345e5a84d1ab39861d9ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae0d7d585f345e5a84d1ab39861d9ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35699 } }
I20260926 20:54:37.590226 19623 sys_catalog.cc:455] T 00000000000000000000000000000000 P bae0d7d585f345e5a84d1ab39861d9ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader bae0d7d585f345e5a84d1ab39861d9ec. Latest consensus state: current_term: 1 leader_uuid: "bae0d7d585f345e5a84d1ab39861d9ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae0d7d585f345e5a84d1ab39861d9ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35699 } } }
I20260926 20:54:37.590319 19623 sys_catalog.cc:458] T 00000000000000000000000000000000 P bae0d7d585f345e5a84d1ab39861d9ec [sys.catalog]: This master's current role is: LEADER
I20260926 20:54:37.590572 19622 sys_catalog.cc:455] T 00000000000000000000000000000000 P bae0d7d585f345e5a84d1ab39861d9ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bae0d7d585f345e5a84d1ab39861d9ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae0d7d585f345e5a84d1ab39861d9ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35699 } } }
I20260926 20:54:37.590693 19632 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:54:37.590716 19622 sys_catalog.cc:458] T 00000000000000000000000000000000 P bae0d7d585f345e5a84d1ab39861d9ec [sys.catalog]: This master's current role is: LEADER
I20260926 20:54:37.590740 19620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:37.584436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48362 (local address 127.0.98.62:35699)
0926 20:54:37.584636 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:37.584644 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:37.584679 (+ 35us) server_negotiation.cc:407] Connection header received
0926 20:54:37.585280 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:37.585289 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:37.585366 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:37.585527 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:37.585760 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:37.587209 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:37.588232 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:37.588866 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:37.588997 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:37.589766 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:37.589771 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:37.589772 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:54:37.590005 (+ 233us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:37.590055 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:54:37.590072 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:37.590152 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:37.590404 (+ 252us) server_negotiation.cc:299] Negotiation successful
0926 20:54:37.590492 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":60,"threads_started":1}
I20260926 20:54:37.591204 19632 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:54:37.592839 19632 catalog_manager.cc:1347] Generated new cluster ID: 557faa9cb6ce4523926ce6ce16108977
I20260926 20:54:37.592900 19632 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:54:37.603907 19632 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:54:37.604447 19632 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:54:37.612628 19632 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bae0d7d585f345e5a84d1ab39861d9ec: Generated new TSK 0
I20260926 20:54:37.612860 19632 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:54:37.619019 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35699
--builtin_ntp_servers=127.0.98.20:33813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 20:54:37.701912 19640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:54:37.702171 19640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:54:37.702214 19640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:54:37.703392 19640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:54:37.703456 19640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:54:37.703506 19640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 20:54:37.705067 19640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:54:37.705349 19640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:54:37.705648 19640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:54:37.711658 19640 server_base.cc:1034] running on GCE node
W20260926 20:54:37.711651 19645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:37.711637 19648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:37.711629 19646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:37.712111 19640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:54:37.712397 19640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:54:37.713553 19640 hybrid_clock.cc:648] HybridClock initialized: now 1790456077713544 us; error 27 us; skew 500 ppm
I20260926 20:54:37.714834 19640 webserver.cc:460] Webserver started at http://127.0.98.1:41677/ using document root <none> and password file <none>
I20260926 20:54:37.715095 19640 fs_manager.cc:359] Metadata directory not provided
I20260926 20:54:37.715143 19640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:54:37.715284 19640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:54:37.716459 19640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/ts-0/data/instance:
uuid: "63da924581084bedb930dcd83975bdc0"
format_stamp: "Formatted at 2026-09-26 20:54:37 on dist-test-slave-hwlj"
server_key: "1b9b931131bba4dd9f4a145b2a3894ef"
server_key_iv: "7098d97aad33ff8fd82cc8f54f4b328a"
server_key_version: "encryptionkey@0"
I20260926 20:54:37.716820 19640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/ts-0/wal/instance:
uuid: "63da924581084bedb930dcd83975bdc0"
format_stamp: "Formatted at 2026-09-26 20:54:37 on dist-test-slave-hwlj"
server_key: "1b9b931131bba4dd9f4a145b2a3894ef"
server_key_iv: "7098d97aad33ff8fd82cc8f54f4b328a"
server_key_version: "encryptionkey@0"
I20260926 20:54:37.718110 19640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:54:37.718870 19654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:37.719065 19640 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:54:37.719162 19640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/ts-0/wal
uuid: "63da924581084bedb930dcd83975bdc0"
format_stamp: "Formatted at 2026-09-26 20:54:37 on dist-test-slave-hwlj"
server_key: "1b9b931131bba4dd9f4a145b2a3894ef"
server_key_iv: "7098d97aad33ff8fd82cc8f54f4b328a"
server_key_version: "encryptionkey@0"
I20260926 20:54:37.719233 19640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:54:37.728713 19640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:54:37.729066 19640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:54:37.729238 19640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:54:37.729497 19640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:54:37.729866 19640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:54:37.729928 19640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:37.729966 19640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:54:37.730014 19640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:37.736809 19640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40021
I20260926 20:54:37.736876 19767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40021 every 8 connection(s)
I20260926 20:54:37.737293 19640 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/ts-0/data/info.pb
I20260926 20:54:37.742761 19620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:37.738111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35851 (local address 127.0.98.62:35699)
0926 20:54:37.738156 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:37.738158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:37.738427 (+ 269us) server_negotiation.cc:407] Connection header received
0926 20:54:37.738993 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:37.738994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:37.739007 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:37.739057 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:37.739788 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:37.740285 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:37.741296 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:37.741497 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:37.741590 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:37.742454 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:37.742455 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:37.742456 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:54:37.742461 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:37.742482 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:54:37.742486 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:37.742540 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:37.742638 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 20:54:37.742668 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 20:54:37.743007 19770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:37.738101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35699 (local address 127.0.98.1:35851)
0926 20:54:37.738306 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:37.738341 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:37.738873 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:37.739131 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:37.739135 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:37.739143 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:54:37.739667 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:37.739672 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:37.740383 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:37.740384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:37.741182 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:37.741185 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:37.741600 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:37.741601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:37.741740 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:37.742302 (+ 562us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:54:37.742302 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:54:37.742332 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:54:37.742356 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:54:37.742554 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:54:37.742559 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:54:37.742563 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:54:37.742641 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 20:54:37.742749 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":96,"threads_started":1}
I20260926 20:54:37.743611 19768 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35699
I20260926 20:54:37.743696 19768 heartbeater.cc:464] Registering TS with master...
I20260926 20:54:37.743932 19768 heartbeater.cc:511] Master 127.0.98.62:35699 requested a full tablet report, sending...
I20260926 20:54:37.744014 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 19640
I20260926 20:54:37.744117 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/ts-0/wal/instance
I20260926 20:54:37.744333 392 external_mini_cluster.cc:1428] Setting key 31b1b93b1b918ef7b5603e710012bec5
I20260926 20:54:37.744503 19580 ts_manager.cc:194] Registered new tserver with Master: 63da924581084bedb930dcd83975bdc0 (127.0.98.1:40021)
I20260926 20:54:37.745396 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35699
--builtin_ntp_servers=127.0.98.20:33813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 20:54:37.745389 19580 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:35851
W20260926 20:54:37.827116 19771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:54:37.827332 19771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:54:37.827354 19771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:54:37.828564 19771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:54:37.828615 19771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:54:37.828670 19771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 20:54:37.830111 19771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:54:37.830346 19771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:54:37.830644 19771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:54:37.835803 19777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:37.835813 19776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:37.835815 19779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:37.835906 19771 server_base.cc:1034] running on GCE node
I20260926 20:54:37.836220 19771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:54:37.836470 19771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:54:37.837606 19771 hybrid_clock.cc:648] HybridClock initialized: now 1790456077837594 us; error 29 us; skew 500 ppm
I20260926 20:54:37.839057 19771 webserver.cc:460] Webserver started at http://127.0.98.2:34837/ using document root <none> and password file <none>
I20260926 20:54:37.839341 19771 fs_manager.cc:359] Metadata directory not provided
I20260926 20:54:37.839416 19771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:54:37.839589 19771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:54:37.841166 19771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/ts-1/data/instance:
uuid: "cc3674c6f7954d988abb1c0f74879b28"
format_stamp: "Formatted at 2026-09-26 20:54:37 on dist-test-slave-hwlj"
server_key: "da73f5f2ebd90b89947df59864785ff2"
server_key_iv: "b05079d537c5f28441ecc76a80afb341"
server_key_version: "encryptionkey@0"
I20260926 20:54:37.841622 19771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/ts-1/wal/instance:
uuid: "cc3674c6f7954d988abb1c0f74879b28"
format_stamp: "Formatted at 2026-09-26 20:54:37 on dist-test-slave-hwlj"
server_key: "da73f5f2ebd90b89947df59864785ff2"
server_key_iv: "b05079d537c5f28441ecc76a80afb341"
server_key_version: "encryptionkey@0"
I20260926 20:54:37.842917 19771 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:54:37.843645 19785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:37.843834 19771 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:54:37.843937 19771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/ts-1/wal
uuid: "cc3674c6f7954d988abb1c0f74879b28"
format_stamp: "Formatted at 2026-09-26 20:54:37 on dist-test-slave-hwlj"
server_key: "da73f5f2ebd90b89947df59864785ff2"
server_key_iv: "b05079d537c5f28441ecc76a80afb341"
server_key_version: "encryptionkey@0"
I20260926 20:54:37.844022 19771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:54:37.859532 19771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:54:37.859906 19771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:54:37.860056 19771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:54:37.860282 19771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:54:37.860697 19771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:54:37.860746 19771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:37.860780 19771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:54:37.860806 19771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:37.866663 19771 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32925
I20260926 20:54:37.866707 19898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32925 every 8 connection(s)
I20260926 20:54:37.867031 19771 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/ts-1/data/info.pb
I20260926 20:54:37.870477 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 19771
I20260926 20:54:37.870585 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/ts-1/wal/instance
I20260926 20:54:37.870826 392 external_mini_cluster.cc:1428] Setting key f059dfd8c1f321a3be57dfb24e5275d8
I20260926 20:54:37.871949 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35699
--builtin_ntp_servers=127.0.98.20:33813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 20:54:37.872471 19620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:37.867713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39083 (local address 127.0.98.62:35699)
0926 20:54:37.867778 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:37.867780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:37.868050 (+ 270us) server_negotiation.cc:407] Connection header received
0926 20:54:37.868551 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:37.868553 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:37.868564 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:37.868618 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:37.869246 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:37.869765 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:37.870759 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:37.871048 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:37.871125 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:37.872143 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:37.872145 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:37.872145 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:54:37.872150 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:37.872165 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:54:37.872170 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:37.872202 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:37.872321 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 20:54:37.872349 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 20:54:37.872563 19901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:37.867738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35699 (local address 127.0.98.2:39083)
0926 20:54:37.867944 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:37.867976 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:37.868462 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:37.868671 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:37.868674 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:37.868683 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:54:37.869166 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:37.869170 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:37.869845 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:37.869845 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:37.870677 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:37.870680 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:37.871128 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:37.871129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:37.871278 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:37.871953 (+ 675us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:54:37.871954 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:54:37.871983 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:54:37.872027 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:54:37.872236 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:54:37.872240 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:54:37.872242 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 20:54:37.872291 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 20:54:37.872381 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
I20260926 20:54:37.873071 19899 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35699
I20260926 20:54:37.873158 19899 heartbeater.cc:464] Registering TS with master...
I20260926 20:54:37.873382 19899 heartbeater.cc:511] Master 127.0.98.62:35699 requested a full tablet report, sending...
I20260926 20:54:37.873783 19580 ts_manager.cc:194] Registered new tserver with Master: cc3674c6f7954d988abb1c0f74879b28 (127.0.98.2:32925)
I20260926 20:54:37.874302 19580 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:39083
W20260926 20:54:37.953006 19902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:54:37.953202 19902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:54:37.953222 19902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:54:37.954262 19902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:54:37.954294 19902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:54:37.954319 19902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 20:54:37.955572 19902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:54:37.955808 19902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:54:37.956060 19902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:54:37.961053 19907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:37.961041 19908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:37.961045 19910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:37.961311 19902 server_base.cc:1034] running on GCE node
I20260926 20:54:37.961583 19902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:54:37.961880 19902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:54:37.963057 19902 hybrid_clock.cc:648] HybridClock initialized: now 1790456077963043 us; error 30 us; skew 500 ppm
I20260926 20:54:37.964457 19902 webserver.cc:460] Webserver started at http://127.0.98.3:43989/ using document root <none> and password file <none>
I20260926 20:54:37.964771 19902 fs_manager.cc:359] Metadata directory not provided
I20260926 20:54:37.964848 19902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:54:37.964993 19902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:54:37.966099 19902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/ts-2/data/instance:
uuid: "71a5dbf551044f59a0b52db81a78ea2b"
format_stamp: "Formatted at 2026-09-26 20:54:37 on dist-test-slave-hwlj"
server_key: "c3c28b0c02cf2e119b0de7f4c3544339"
server_key_iv: "9e56579e46a801cf278c24a86935a591"
server_key_version: "encryptionkey@0"
I20260926 20:54:37.966450 19902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/ts-2/wal/instance:
uuid: "71a5dbf551044f59a0b52db81a78ea2b"
format_stamp: "Formatted at 2026-09-26 20:54:37 on dist-test-slave-hwlj"
server_key: "c3c28b0c02cf2e119b0de7f4c3544339"
server_key_iv: "9e56579e46a801cf278c24a86935a591"
server_key_version: "encryptionkey@0"
I20260926 20:54:37.967833 19902 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:54:37.968581 19916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:37.968854 19902 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:54:37.968955 19902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/ts-2/wal
uuid: "71a5dbf551044f59a0b52db81a78ea2b"
format_stamp: "Formatted at 2026-09-26 20:54:37 on dist-test-slave-hwlj"
server_key: "c3c28b0c02cf2e119b0de7f4c3544339"
server_key_iv: "9e56579e46a801cf278c24a86935a591"
server_key_version: "encryptionkey@0"
I20260926 20:54:37.969029 19902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:54:37.997052 19902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:54:37.997383 19902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:54:37.997536 19902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:54:37.997778 19902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:54:37.998104 19902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:54:37.998155 19902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:37.998198 19902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:54:37.998221 19902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:38.003665 19902 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43911
I20260926 20:54:38.003739 20029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43911 every 8 connection(s)
I20260926 20:54:38.004091 19902 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/ts-2/data/info.pb
I20260926 20:54:38.007813 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 19902
I20260926 20:54:38.007913 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/ts-2/wal/instance
I20260926 20:54:38.008105 392 external_mini_cluster.cc:1428] Setting key e9e8a12628e5043bb127cddee97e6913
I20260926 20:54:38.009346 19620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:38.004916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54727 (local address 127.0.98.62:35699)
0926 20:54:38.004990 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:38.004992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:38.005201 (+ 209us) server_negotiation.cc:407] Connection header received
0926 20:54:38.005670 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:38.005671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:38.005681 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:38.005724 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:38.006345 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:38.006817 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:38.007865 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:38.008107 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:38.008197 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:38.009091 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:38.009092 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:38.009093 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:54:38.009098 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:38.009111 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:54:38.009115 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:38.009170 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:38.009245 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 20:54:38.009270 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 20:54:38.009557 20032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:38.004892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35699 (local address 127.0.98.3:54727)
0926 20:54:38.005084 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:38.005118 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:38.005580 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:38.005778 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:38.005781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:38.005789 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:54:38.006279 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:38.006283 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:38.006892 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:38.006893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:38.007744 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:38.007746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:38.008209 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:38.008210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:38.008349 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:38.008952 (+ 603us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:54:38.008953 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:54:38.008981 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:54:38.009002 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:54:38.009173 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:54:38.009178 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:54:38.009181 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:54:38.009248 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 20:54:38.009347 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":98,"threads_started":1}
I20260926 20:54:38.009963 20030 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35699
I20260926 20:54:38.010067 20030 heartbeater.cc:464] Registering TS with master...
I20260926 20:54:38.010238 20030 heartbeater.cc:511] Master 127.0.98.62:35699 requested a full tablet report, sending...
I20260926 20:54:38.010581 19580 ts_manager.cc:194] Registered new tserver with Master: 71a5dbf551044f59a0b52db81a78ea2b (127.0.98.3:43911)
I20260926 20:54:38.010942 19580 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:54727
I20260926 20:54:38.019922 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:54:38.024309 19620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:38.021363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48372 (local address 127.0.98.62:35699)
0926 20:54:38.021408 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:38.021410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:38.021486 (+ 76us) server_negotiation.cc:407] Connection header received
0926 20:54:38.021561 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:38.021562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:38.021572 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:38.021623 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:38.021791 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:38.022302 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:38.023096 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:38.023308 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:38.023388 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:38.024073 (+ 685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:38.024075 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:38.024075 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:54:38.024080 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:38.024093 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:54:38.024097 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:38.024144 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:38.024191 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 20:54:38.024214 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 20:54:38.025878 19580 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48372:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:54:38.027439 19580 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:54:38.033564 20040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:38.029593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43718 (local address 127.0.98.2:32925)
0926 20:54:38.029961 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:38.029965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:38.029973 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:54:38.030006 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:38.030007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:38.030036 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:38.030116 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:54:38.030256 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:38.031105 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:38.032536 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:38.033180 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:38.033269 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:38.033371 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:38.033444 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 20:54:38.033480 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":70,"threads_started":1}
I20260926 20:54:38.033553 20038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:38.029514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32925 (local address 127.0.0.1:43718)
0926 20:54:38.029716 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:38.029757 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:38.029823 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:38.030120 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:38.030122 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:38.030124 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:54:38.030176 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:38.030177 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:38.031186 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:38.031187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:38.032461 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:38.032464 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:38.033272 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:38.033273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:38.033344 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:38.033356 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 20:54:38.033431 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 20:54:38.033481 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":84,"threads_started":1}
I20260926 20:54:38.033833 20042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:38.029738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43911 (local address 127.0.0.1:46828)
0926 20:54:38.029860 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:38.029870 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:38.029921 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:38.030300 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:38.030302 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:38.030304 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:54:38.030346 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:38.030348 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:38.031607 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:38.031608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:38.032676 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:38.032678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:38.033584 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:38.033585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:38.033644 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:38.033649 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:54:38.033707 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 20:54:38.033749 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":26,"thread_start_us":75,"threads_started":1}
I20260926 20:54:38.034094 20043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:38.029939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46828 (local address 127.0.98.3:43911)
0926 20:54:38.030108 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:38.030112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:38.030120 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:54:38.030153 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:38.030155 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:38.030169 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:38.030239 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:54:38.030425 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:38.031521 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:38.032754 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:38.033399 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:38.033469 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:38.033549 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:38.033978 (+ 429us) server_negotiation.cc:299] Negotiation successful
0926 20:54:38.034018 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":77,"threads_started":1}
I20260926 20:54:38.034303 19833 tablet_service.cc:1467] Processing CreateTablet for tablet 87def31f5d8b4096a086d1f1342a3d8b (DEFAULT_TABLE table=test-table [id=36ee6e961f3c468f8250acdc1a194a9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:54:38.034497 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:38.029627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34884 (local address 127.0.98.1:40021)
0926 20:54:38.030369 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:38.030373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:38.030382 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:54:38.030422 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:38.030424 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:38.030438 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:38.030541 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:54:38.030645 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:38.031687 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:38.032936 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:38.034064 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:38.034135 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:38.034321 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:38.034352 (+ 31us) server_negotiation.cc:299] Negotiation successful
0926 20:54:38.034394 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":696,"thread_start_us":81,"threads_started":1}
I20260926 20:54:38.034715 19833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87def31f5d8b4096a086d1f1342a3d8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:54:38.034706 19964 tablet_service.cc:1467] Processing CreateTablet for tablet 87def31f5d8b4096a086d1f1342a3d8b (DEFAULT_TABLE table=test-table [id=36ee6e961f3c468f8250acdc1a194a9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:54:38.035029 19964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87def31f5d8b4096a086d1f1342a3d8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:54:38.035993 20039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:38.029519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40021 (local address 127.0.0.1:34884)
0926 20:54:38.029744 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:38.029763 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:38.029823 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:38.030547 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:38.030547 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:38.030550 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:54:38.030586 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:38.030587 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:38.031785 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:38.031786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:38.032852 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:38.032855 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:38.034143 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:38.034144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:38.034281 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:38.034286 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:54:38.035904 (+ 1618us) client_negotiation.cc:240] Negotiation successful
0926 20:54:38.035944 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":1,"spinlock_wait_cycles":384,"thread_start_us":131,"threads_started":1}
I20260926 20:54:38.038233 20044 tablet_bootstrap.cc:492] T 87def31f5d8b4096a086d1f1342a3d8b P 71a5dbf551044f59a0b52db81a78ea2b: Bootstrap starting.
I20260926 20:54:38.038987 20044 tablet_bootstrap.cc:654] T 87def31f5d8b4096a086d1f1342a3d8b P 71a5dbf551044f59a0b52db81a78ea2b: Neither blocks nor log segments found. Creating new log.
I20260926 20:54:38.038933 19702 tablet_service.cc:1467] Processing CreateTablet for tablet 87def31f5d8b4096a086d1f1342a3d8b (DEFAULT_TABLE table=test-table [id=36ee6e961f3c468f8250acdc1a194a9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:54:38.039317 20044 log.cc:824] T 87def31f5d8b4096a086d1f1342a3d8b P 71a5dbf551044f59a0b52db81a78ea2b: Log is configured to *not* fsync() on all Append() calls
I20260926 20:54:38.039336 19702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87def31f5d8b4096a086d1f1342a3d8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:54:38.039675 20045 tablet_bootstrap.cc:492] T 87def31f5d8b4096a086d1f1342a3d8b P cc3674c6f7954d988abb1c0f74879b28: Bootstrap starting.
I20260926 20:54:38.040083 20044 tablet_bootstrap.cc:492] T 87def31f5d8b4096a086d1f1342a3d8b P 71a5dbf551044f59a0b52db81a78ea2b: No bootstrap required, opened a new log
I20260926 20:54:38.040160 20044 ts_tablet_manager.cc:1397] T 87def31f5d8b4096a086d1f1342a3d8b P 71a5dbf551044f59a0b52db81a78ea2b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:54:38.040355 20045 tablet_bootstrap.cc:654] T 87def31f5d8b4096a086d1f1342a3d8b P cc3674c6f7954d988abb1c0f74879b28: Neither blocks nor log segments found. Creating new log.
I20260926 20:54:38.040664 20045 log.cc:824] T 87def31f5d8b4096a086d1f1342a3d8b P cc3674c6f7954d988abb1c0f74879b28: Log is configured to *not* fsync() on all Append() calls
I20260926 20:54:38.041420 20045 tablet_bootstrap.cc:492] T 87def31f5d8b4096a086d1f1342a3d8b P cc3674c6f7954d988abb1c0f74879b28: No bootstrap required, opened a new log
I20260926 20:54:38.041495 20045 ts_tablet_manager.cc:1397] T 87def31f5d8b4096a086d1f1342a3d8b P cc3674c6f7954d988abb1c0f74879b28: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 20:54:38.042238 20044 raft_consensus.cc:348] T 87def31f5d8b4096a086d1f1342a3d8b P 71a5dbf551044f59a0b52db81a78ea2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc3674c6f7954d988abb1c0f74879b28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32925 } } peers { permanent_uuid: "63da924581084bedb930dcd83975bdc0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40021 } } peers { permanent_uuid: "71a5dbf551044f59a0b52db81a78ea2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43911 } }
I20260926 20:54:38.042388 20044 raft_consensus.cc:374] T 87def31f5d8b4096a086d1f1342a3d8b P 71a5dbf551044f59a0b52db81a78ea2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:54:38.042433 20044 raft_consensus.cc:729] T 87def31f5d8b4096a086d1f1342a3d8b P 71a5dbf551044f59a0b52db81a78ea2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a5dbf551044f59a0b52db81a78ea2b, State: Initialized, Role: FOLLOWER
I20260926 20:54:38.042567 20044 consensus_queue.cc:260] T 87def31f5d8b4096a086d1f1342a3d8b P 71a5dbf551044f59a0b52db81a78ea2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc3674c6f7954d988abb1c0f74879b28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32925 } } peers { permanent_uuid: "63da924581084bedb930dcd83975bdc0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40021 } } peers { permanent_uuid: "71a5dbf551044f59a0b52db81a78ea2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43911 } }
I20260926 20:54:38.042884 20049 tablet_bootstrap.cc:492] T 87def31f5d8b4096a086d1f1342a3d8b P 63da924581084bedb930dcd83975bdc0: Bootstrap starting.
I20260926 20:54:38.042835 20044 ts_tablet_manager.cc:1428] T 87def31f5d8b4096a086d1f1342a3d8b P 71a5dbf551044f59a0b52db81a78ea2b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:54:38.043097 20030 heartbeater.cc:503] Master 127.0.98.62:35699 was elected leader, sending a full tablet report...
I20260926 20:54:38.043637 20049 tablet_bootstrap.cc:654] T 87def31f5d8b4096a086d1f1342a3d8b P 63da924581084bedb930dcd83975bdc0: Neither blocks nor log segments found. Creating new log.
I20260926 20:54:38.043982 20049 log.cc:824] T 87def31f5d8b4096a086d1f1342a3d8b P 63da924581084bedb930dcd83975bdc0: Log is configured to *not* fsync() on all Append() calls
I20260926 20:54:38.044116 20045 raft_consensus.cc:348] T 87def31f5d8b4096a086d1f1342a3d8b P cc3674c6f7954d988abb1c0f74879b28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc3674c6f7954d988abb1c0f74879b28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32925 } } peers { permanent_uuid: "63da924581084bedb930dcd83975bdc0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40021 } } peers { permanent_uuid: "71a5dbf551044f59a0b52db81a78ea2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43911 } }
I20260926 20:54:38.044293 20045 raft_consensus.cc:374] T 87def31f5d8b4096a086d1f1342a3d8b P cc3674c6f7954d988abb1c0f74879b28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:54:38.044348 20045 raft_consensus.cc:729] T 87def31f5d8b4096a086d1f1342a3d8b P cc3674c6f7954d988abb1c0f74879b28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc3674c6f7954d988abb1c0f74879b28, State: Initialized, Role: FOLLOWER
I20260926 20:54:38.044473 20045 consensus_queue.cc:260] T 87def31f5d8b4096a086d1f1342a3d8b P cc3674c6f7954d988abb1c0f74879b28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc3674c6f7954d988abb1c0f74879b28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32925 } } peers { permanent_uuid: "63da924581084bedb930dcd83975bdc0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40021 } } peers { permanent_uuid: "71a5dbf551044f59a0b52db81a78ea2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43911 } }
I20260926 20:54:38.044797 20049 tablet_bootstrap.cc:492] T 87def31f5d8b4096a086d1f1342a3d8b P 63da924581084bedb930dcd83975bdc0: No bootstrap required, opened a new log
I20260926 20:54:38.044790 20045 ts_tablet_manager.cc:1428] T 87def31f5d8b4096a086d1f1342a3d8b P cc3674c6f7954d988abb1c0f74879b28: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260926 20:54:38.044809 19899 heartbeater.cc:503] Master 127.0.98.62:35699 was elected leader, sending a full tablet report...
I20260926 20:54:38.044863 20049 ts_tablet_manager.cc:1397] T 87def31f5d8b4096a086d1f1342a3d8b P 63da924581084bedb930dcd83975bdc0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:54:38.047008 20049 raft_consensus.cc:348] T 87def31f5d8b4096a086d1f1342a3d8b P 63da924581084bedb930dcd83975bdc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc3674c6f7954d988abb1c0f74879b28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32925 } } peers { permanent_uuid: "63da924581084bedb930dcd83975bdc0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40021 } } peers { permanent_uuid: "71a5dbf551044f59a0b52db81a78ea2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43911 } }
I20260926 20:54:38.047135 20049 raft_consensus.cc:374] T 87def31f5d8b4096a086d1f1342a3d8b P 63da924581084bedb930dcd83975bdc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:54:38.047158 20049 raft_consensus.cc:729] T 87def31f5d8b4096a086d1f1342a3d8b P 63da924581084bedb930dcd83975bdc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63da924581084bedb930dcd83975bdc0, State: Initialized, Role: FOLLOWER
I20260926 20:54:38.047232 20049 consensus_queue.cc:260] T 87def31f5d8b4096a086d1f1342a3d8b P 63da924581084bedb930dcd83975bdc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc3674c6f7954d988abb1c0f74879b28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32925 } } peers { permanent_uuid: "63da924581084bedb930dcd83975bdc0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40021 } } peers { permanent_uuid: "71a5dbf551044f59a0b52db81a78ea2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43911 } }
I20260926 20:54:38.047430 20049 ts_tablet_manager.cc:1428] T 87def31f5d8b4096a086d1f1342a3d8b P 63da924581084bedb930dcd83975bdc0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:54:38.047514 19768 heartbeater.cc:503] Master 127.0.98.62:35699 was elected leader, sending a full tablet report...
W20260926 20:54:38.117615 19900 tablet.cc:2369] T 87def31f5d8b4096a086d1f1342a3d8b P cc3674c6f7954d988abb1c0f74879b28: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:54:38.238119 19769 tablet.cc:2369] T 87def31f5d8b4096a086d1f1342a3d8b P 63da924581084bedb930dcd83975bdc0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:54:38.254669 20031 tablet.cc:2369] T 87def31f5d8b4096a086d1f1342a3d8b P 71a5dbf551044f59a0b52db81a78ea2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:54:38.485967 20051 raft_consensus.cc:482] T 87def31f5d8b4096a086d1f1342a3d8b P cc3674c6f7954d988abb1c0f74879b28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:54:38.486126 20051 raft_consensus.cc:504] T 87def31f5d8b4096a086d1f1342a3d8b P cc3674c6f7954d988abb1c0f74879b28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc3674c6f7954d988abb1c0f74879b28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32925 } } peers { permanent_uuid: "63da924581084bedb930dcd83975bdc0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40021 } } peers { permanent_uuid: "71a5dbf551044f59a0b52db81a78ea2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43911 } }
I20260926 20:54:38.486523 20051 leader_election.cc:288] T 87def31f5d8b4096a086d1f1342a3d8b P cc3674c6f7954d988abb1c0f74879b28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63da924581084bedb930dcd83975bdc0 (127.0.98.1:40021), 71a5dbf551044f59a0b52db81a78ea2b (127.0.98.3:43911)
I20260926 20:54:38.489461 20043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:38.486830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49051 (local address 127.0.98.3:43911)
0926 20:54:38.486875 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:38.486878 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:38.486912 (+ 34us) server_negotiation.cc:407] Connection header received
0926 20:54:38.487006 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:38.487007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:38.487016 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:38.487087 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:54:38.487232 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:38.487696 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:38.488773 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:38.489125 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:38.489203 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:38.489213 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:38.489355 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 20:54:38.489390 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 20:54:38.489475 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:38.486713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43911 (local address 127.0.98.2:49051)
0926 20:54:38.486847 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:38.486859 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:38.486928 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:38.487092 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:38.487092 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:38.487094 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:54:38.487166 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:38.487167 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:38.487799 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:38.487801 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:38.488692 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:38.488695 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:38.489203 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:38.489204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:38.489273 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:38.489277 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:54:38.489350 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 20:54:38.489389 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":21,"thread_start_us":67,"threads_started":1}
I20260926 20:54:38.489545 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:38.486781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35919 (local address 127.0.98.1:40021)
0926 20:54:38.486831 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:38.486833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:38.486944 (+ 111us) server_negotiation.cc:407] Connection header received
0926 20:54:38.487024 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:38.487025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:38.487036 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:38.487094 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:54:38.487282 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:38.487731 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:38.488715 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:38.489166 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:38.489242 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:38.489253 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:38.489438 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 20:54:38.489476 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 20:54:38.489557 20053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:38.486720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40021 (local address 127.0.98.2:35919)
0926 20:54:38.486865 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:38.486875 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:38.486954 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:38.487096 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:38.487097 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:38.487099 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:54:38.487194 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:38.487195 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:38.487852 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:38.487853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:38.488616 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:38.488618 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:38.489252 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:38.489253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:38.489323 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:38.489326 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:54:38.489400 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 20:54:38.489441 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":20,"thread_start_us":70,"threads_started":1}
I20260926 20:54:38.489866 19722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87def31f5d8b4096a086d1f1342a3d8b" candidate_uuid: "cc3674c6f7954d988abb1c0f74879b28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63da924581084bedb930dcd83975bdc0" is_pre_election: true
I20260926 20:54:38.489972 19984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87def31f5d8b4096a086d1f1342a3d8b" candidate_uuid: "cc3674c6f7954d988abb1c0f74879b28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a5dbf551044f59a0b52db81a78ea2b" is_pre_election: true
I20260926 20:54:38.490054 19722 raft_consensus.cc:2454] T 87def31f5d8b4096a086d1f1342a3d8b P 63da924581084bedb930dcd83975bdc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc3674c6f7954d988abb1c0f74879b28 in term 0.
I20260926 20:54:38.490118 19984 raft_consensus.cc:2454] T 87def31f5d8b4096a086d1f1342a3d8b P 71a5dbf551044f59a0b52db81a78ea2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc3674c6f7954d988abb1c0f74879b28 in term 0.
I20260926 20:54:38.490288 19786 leader_election.cc:302] T 87def31f5d8b4096a086d1f1342a3d8b P cc3674c6f7954d988abb1c0f74879b28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63da924581084bedb930dcd83975bdc0, cc3674c6f7954d988abb1c0f74879b28; no voters:
I20260926 20:54:38.490438 20051 raft_consensus.cc:2781] T 87def31f5d8b4096a086d1f1342a3d8b P cc3674c6f7954d988abb1c0f74879b28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:54:38.490494 20051 raft_consensus.cc:482] T 87def31f5d8b4096a086d1f1342a3d8b P cc3674c6f7954d988abb1c0f74879b28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:54:38.490521 20051 raft_consensus.cc:3037] T 87def31f5d8b4096a086d1f1342a3d8b P cc3674c6f7954d988abb1c0f74879b28 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:54:38.491338 20051 raft_consensus.cc:504] T 87def31f5d8b4096a086d1f1342a3d8b P cc3674c6f7954d988abb1c0f74879b28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc3674c6f7954d988abb1c0f74879b28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32925 } } peers { permanent_uuid: "63da924581084bedb930dcd83975bdc0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40021 } } peers { permanent_uuid: "71a5dbf551044f59a0b52db81a78ea2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43911 } }
I20260926 20:54:38.491483 20051 leader_election.cc:288] T 87def31f5d8b4096a086d1f1342a3d8b P cc3674c6f7954d988abb1c0f74879b28 [CANDIDATE]: Term 1 election: Requested vote from peers 63da924581084bedb930dcd83975bdc0 (127.0.98.1:40021), 71a5dbf551044f59a0b52db81a78ea2b (127.0.98.3:43911)
I20260926 20:54:38.491626 19722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87def31f5d8b4096a086d1f1342a3d8b" candidate_uuid: "cc3674c6f7954d988abb1c0f74879b28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63da924581084bedb930dcd83975bdc0"
I20260926 20:54:38.491680 19984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87def31f5d8b4096a086d1f1342a3d8b" candidate_uuid: "cc3674c6f7954d988abb1c0f74879b28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a5dbf551044f59a0b52db81a78ea2b"
I20260926 20:54:38.491782 19984 raft_consensus.cc:3037] T 87def31f5d8b4096a086d1f1342a3d8b P 71a5dbf551044f59a0b52db81a78ea2b [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:54:38.491729 19722 raft_consensus.cc:3037] T 87def31f5d8b4096a086d1f1342a3d8b P 63da924581084bedb930dcd83975bdc0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:54:38.492492 19984 raft_consensus.cc:2454] T 87def31f5d8b4096a086d1f1342a3d8b P 71a5dbf551044f59a0b52db81a78ea2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc3674c6f7954d988abb1c0f74879b28 in term 1.
I20260926 20:54:38.492499 19722 raft_consensus.cc:2454] T 87def31f5d8b4096a086d1f1342a3d8b P 63da924581084bedb930dcd83975bdc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc3674c6f7954d988abb1c0f74879b28 in term 1.
I20260926 20:54:38.492660 19787 leader_election.cc:302] T 87def31f5d8b4096a086d1f1342a3d8b P cc3674c6f7954d988abb1c0f74879b28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71a5dbf551044f59a0b52db81a78ea2b, cc3674c6f7954d988abb1c0f74879b28; no voters:
I20260926 20:54:38.492759 20051 raft_consensus.cc:2781] T 87def31f5d8b4096a086d1f1342a3d8b P cc3674c6f7954d988abb1c0f74879b28 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:54:38.492898 20051 raft_consensus.cc:686] T 87def31f5d8b4096a086d1f1342a3d8b P cc3674c6f7954d988abb1c0f74879b28 [term 1 LEADER]: Becoming Leader. State: Replica: cc3674c6f7954d988abb1c0f74879b28, State: Running, Role: LEADER
I20260926 20:54:38.493000 20051 consensus_queue.cc:237] T 87def31f5d8b4096a086d1f1342a3d8b P cc3674c6f7954d988abb1c0f74879b28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc3674c6f7954d988abb1c0f74879b28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32925 } } peers { permanent_uuid: "63da924581084bedb930dcd83975bdc0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40021 } } peers { permanent_uuid: "71a5dbf551044f59a0b52db81a78ea2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43911 } }
I20260926 20:54:38.493865 19579 catalog_manager.cc:5495] T 87def31f5d8b4096a086d1f1342a3d8b P cc3674c6f7954d988abb1c0f74879b28 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc3674c6f7954d988abb1c0f74879b28 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "cc3674c6f7954d988abb1c0f74879b28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc3674c6f7954d988abb1c0f74879b28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63da924581084bedb930dcd83975bdc0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71a5dbf551044f59a0b52db81a78ea2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43911 } health_report { overall_health: UNKNOWN } } }
I20260926 20:54:38.582409 20059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:38.579668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43720 (local address 127.0.98.2:32925)
0926 20:54:38.579802 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:38.579805 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:38.579894 (+ 89us) server_negotiation.cc:407] Connection header received
0926 20:54:38.579944 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:38.579945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:38.579968 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:38.580077 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:54:38.580208 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:38.580940 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:38.581711 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:38.581951 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:38.582008 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:38.582147 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:54:38.582244 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:54:38.582302 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:38.582330 (+ 28us) server_negotiation.cc:299] Negotiation successful
0926 20:54:38.582348 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":48,"threads_started":1}
I20260926 20:54:38.584231 19722 raft_consensus.cc:1261] T 87def31f5d8b4096a086d1f1342a3d8b P 63da924581084bedb930dcd83975bdc0 [term 1 FOLLOWER]: Refusing update from remote peer cc3674c6f7954d988abb1c0f74879b28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:54:38.584227 19984 raft_consensus.cc:1261] T 87def31f5d8b4096a086d1f1342a3d8b P 71a5dbf551044f59a0b52db81a78ea2b [term 1 FOLLOWER]: Refusing update from remote peer cc3674c6f7954d988abb1c0f74879b28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:54:38.584602 20055 consensus_queue.cc:1035] T 87def31f5d8b4096a086d1f1342a3d8b P cc3674c6f7954d988abb1c0f74879b28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71a5dbf551044f59a0b52db81a78ea2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:54:38.584816 20051 consensus_queue.cc:1035] T 87def31f5d8b4096a086d1f1342a3d8b P cc3674c6f7954d988abb1c0f74879b28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63da924581084bedb930dcd83975bdc0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:54:38.587376 20064 mvcc.cc:205] Tried to move back new op lower bound from 7333708097877934080 to 7333708097508044800. Current Snapshot: MvccSnapshot[applied={T|T < 7333708097877934080}]
I20260926 20:54:38.587773 20063 mvcc.cc:205] Tried to move back new op lower bound from 7333708097877934080 to 7333708097508044800. Current Snapshot: MvccSnapshot[applied={T|T < 7333708097877934080 or (T in {7333708097877934080})}]
I20260926 20:54:38.590166 19579 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:54:38.590276 19579 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:54:38.591034 19579 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bae0d7d585f345e5a84d1ab39861d9ec: Sending DeleteTablet for 3 replicas of tablet 87def31f5d8b4096a086d1f1342a3d8b
I20260926 20:54:38.591353 19833 tablet_service.cc:1514] Processing DeleteTablet for tablet 87def31f5d8b4096a086d1f1342a3d8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:54:38 UTC) from {username='slave'} at 127.0.0.1:43718
I20260926 20:54:38.591355 19702 tablet_service.cc:1514] Processing DeleteTablet for tablet 87def31f5d8b4096a086d1f1342a3d8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:54:38 UTC) from {username='slave'} at 127.0.0.1:34884
I20260926 20:54:38.591555 20072 tablet_replica.cc:330] T 87def31f5d8b4096a086d1f1342a3d8b P cc3674c6f7954d988abb1c0f74879b28: stopping tablet replica
I20260926 20:54:38.591650 20072 raft_consensus.cc:2229] T 87def31f5d8b4096a086d1f1342a3d8b P cc3674c6f7954d988abb1c0f74879b28 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:54:38.591660 19964 tablet_service.cc:1514] Processing DeleteTablet for tablet 87def31f5d8b4096a086d1f1342a3d8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:54:38 UTC) from {username='slave'} at 127.0.0.1:46828
I20260926 20:54:38.591813 20073 tablet_replica.cc:330] T 87def31f5d8b4096a086d1f1342a3d8b P 63da924581084bedb930dcd83975bdc0: stopping tablet replica
I20260926 20:54:38.591835 20072 raft_consensus.cc:2258] T 87def31f5d8b4096a086d1f1342a3d8b P cc3674c6f7954d988abb1c0f74879b28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:54:38.591964 20073 raft_consensus.cc:2229] T 87def31f5d8b4096a086d1f1342a3d8b P 63da924581084bedb930dcd83975bdc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:54:38.592212 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 19640
I20260926 20:54:38.592113 20073 raft_consensus.cc:2258] T 87def31f5d8b4096a086d1f1342a3d8b P 63da924581084bedb930dcd83975bdc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:54:38.592255 20072 ts_tablet_manager.cc:1905] T 87def31f5d8b4096a086d1f1342a3d8b P cc3674c6f7954d988abb1c0f74879b28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:54:38.594678 20072 ts_tablet_manager.cc:1918] T 87def31f5d8b4096a086d1f1342a3d8b P cc3674c6f7954d988abb1c0f74879b28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:54:38.594761 20072 log.cc:1196] T 87def31f5d8b4096a086d1f1342a3d8b P cc3674c6f7954d988abb1c0f74879b28: Deleting WAL directory at /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455725523895-392-0/minicluster-data/ts-1/wal/wals/87def31f5d8b4096a086d1f1342a3d8b
I20260926 20:54:38.595028 20072 ts_tablet_manager.cc:1939] T 87def31f5d8b4096a086d1f1342a3d8b P cc3674c6f7954d988abb1c0f74879b28: Deleting consensus metadata
I20260926 20:54:38.595631 19565 catalog_manager.cc:4841] TS cc3674c6f7954d988abb1c0f74879b28 (127.0.98.2:32925): tablet 87def31f5d8b4096a086d1f1342a3d8b (table test-table [id=36ee6e961f3c468f8250acdc1a194a9d]) successfully deleted
W20260926 20:54:38.599078 19564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40021 (error 108)
I20260926 20:54:38.599195 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 19771
I20260926 20:54:38.600875 20074 tablet_replica.cc:330] T 87def31f5d8b4096a086d1f1342a3d8b P 71a5dbf551044f59a0b52db81a78ea2b: stopping tablet replica
I20260926 20:54:38.601004 20074 raft_consensus.cc:2229] T 87def31f5d8b4096a086d1f1342a3d8b P 71a5dbf551044f59a0b52db81a78ea2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:54:38.601147 20074 raft_consensus.cc:2258] T 87def31f5d8b4096a086d1f1342a3d8b P 71a5dbf551044f59a0b52db81a78ea2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:54:38.601239 20076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:38.601042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40021 (local address 127.0.0.1:34898)
0926 20:54:38.601149 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:38.601197 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":42,"threads_started":1}
W20260926 20:54:38.601356 19564 catalog_manager.cc:4584] TS 63da924581084bedb930dcd83975bdc0 (127.0.98.1:40021): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 87def31f5d8b4096a086d1f1342a3d8b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40021: connect: Connection refused (error 111)
I20260926 20:54:38.604547 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 19902
I20260926 20:54:38.604599 20074 ts_tablet_manager.cc:1905] T 87def31f5d8b4096a086d1f1342a3d8b P 71a5dbf551044f59a0b52db81a78ea2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:54:38.608917 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 19549
I20260926 20:54:38.608949 20076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:38.608808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43911 (local address 127.0.0.1:46842)
0926 20:54:38.608883 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:38.608916 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.3:43911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
2026-09-26T20:54:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1176 ms)
[----------] 6 tests from EncryptionPolicyTest (8513 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455725523895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:54:38 dist-test-slave-hwlj krb5kdc[20079](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:54:38 dist-test-slave-hwlj krb5kdc[20079](info): set up 2 sockets
Sep 26 20:54:38 dist-test-slave-hwlj krb5kdc[20079](info): commencing operation
krb5kdc: starting...
W20260926 20:54:39.640130 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455725523895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455725523895-392-0/krb5kdc/test-user.keytab.
Sep 26 20:54:39 dist-test-slave-hwlj krb5kdc[20079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:54:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:54:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455725523895-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455725523895-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:54:39.762367 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39807
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455725523895-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39807
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:54:39.845710 20092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:54:39.845961 20092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:54:39.846004 20092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:54:39.847424 20092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:54:39.847473 20092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:54:39.847499 20092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:54:39.847522 20092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:54:39.847553 20092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:54:39.850219 20092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455725523895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39807
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455725523895-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:54:39.850497 20092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:54:39.850741 20092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:54:39.855842 20098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:39.855866 20092 server_base.cc:1034] running on GCE node
W20260926 20:54:39.855867 20097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:39.855849 20100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:39.856295 20092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:54:39.856585 20092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:54:39.857776 20092 hybrid_clock.cc:648] HybridClock initialized: now 1790456079857758 us; error 46 us; skew 500 ppm
Sep 26 20:54:39 dist-test-slave-hwlj krb5kdc[20079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:54:39.860553 20092 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:54:39.860991 20092 webserver.cc:460] Webserver started at http://127.0.0.1:40385/ using document root <none> and password file <none>
I20260926 20:54:39.861220 20092 fs_manager.cc:359] Metadata directory not provided
I20260926 20:54:39.861285 20092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:54:39.861431 20092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:54:39.862493 20092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455725523895-392-0/minicluster-data/master-0/data/instance:
uuid: "054d424d4d37445a922616941cca4b36"
format_stamp: "Formatted at 2026-09-26 20:54:39 on dist-test-slave-hwlj"
server_key: "3955e61f35bd411366f4f3fa5344e7d8"
server_key_iv: "7b816932f4a8b3967846201fc8fbad10"
server_key_version: "encryptionkey@0"
I20260926 20:54:39.862855 20092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455725523895-392-0/minicluster-data/master-0/wal/instance:
uuid: "054d424d4d37445a922616941cca4b36"
format_stamp: "Formatted at 2026-09-26 20:54:39 on dist-test-slave-hwlj"
server_key: "3955e61f35bd411366f4f3fa5344e7d8"
server_key_iv: "7b816932f4a8b3967846201fc8fbad10"
server_key_version: "encryptionkey@0"
I20260926 20:54:39.864261 20092 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:54:39.865168 20107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:39.865347 20092 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:54:39.865455 20092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "054d424d4d37445a922616941cca4b36"
format_stamp: "Formatted at 2026-09-26 20:54:39 on dist-test-slave-hwlj"
server_key: "3955e61f35bd411366f4f3fa5344e7d8"
server_key_iv: "7b816932f4a8b3967846201fc8fbad10"
server_key_version: "encryptionkey@0"
I20260926 20:54:39.865530 20092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:54:39.890424 20092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:54:39.892876 20092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:54:39.893040 20092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:54:39.897433 20092 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39807
I20260926 20:54:39.897483 20159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39807 every 8 connection(s)
I20260926 20:54:39.897872 20092 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:54:39.898730 20160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:54:39.900943 20160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 054d424d4d37445a922616941cca4b36: Bootstrap starting.
I20260926 20:54:39.901469 20160 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 054d424d4d37445a922616941cca4b36: Neither blocks nor log segments found. Creating new log.
I20260926 20:54:39.901695 20160 log.cc:824] T 00000000000000000000000000000000 P 054d424d4d37445a922616941cca4b36: Log is configured to *not* fsync() on all Append() calls
I20260926 20:54:39.902329 20160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 054d424d4d37445a922616941cca4b36: No bootstrap required, opened a new log
I20260926 20:54:39.903923 20160 raft_consensus.cc:348] T 00000000000000000000000000000000 P 054d424d4d37445a922616941cca4b36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "054d424d4d37445a922616941cca4b36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39807 } }
I20260926 20:54:39.904057 20160 raft_consensus.cc:374] T 00000000000000000000000000000000 P 054d424d4d37445a922616941cca4b36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:54:39.904078 20160 raft_consensus.cc:729] T 00000000000000000000000000000000 P 054d424d4d37445a922616941cca4b36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 054d424d4d37445a922616941cca4b36, State: Initialized, Role: FOLLOWER
I20260926 20:54:39.904210 20160 consensus_queue.cc:260] T 00000000000000000000000000000000 P 054d424d4d37445a922616941cca4b36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "054d424d4d37445a922616941cca4b36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39807 } }
I20260926 20:54:39.904289 20160 raft_consensus.cc:388] T 00000000000000000000000000000000 P 054d424d4d37445a922616941cca4b36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:54:39.904310 20160 raft_consensus.cc:482] T 00000000000000000000000000000000 P 054d424d4d37445a922616941cca4b36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:54:39.904345 20160 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 054d424d4d37445a922616941cca4b36 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:54:39.904845 20160 raft_consensus.cc:504] T 00000000000000000000000000000000 P 054d424d4d37445a922616941cca4b36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "054d424d4d37445a922616941cca4b36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39807 } }
I20260926 20:54:39.904932 20160 leader_election.cc:302] T 00000000000000000000000000000000 P 054d424d4d37445a922616941cca4b36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 054d424d4d37445a922616941cca4b36; no voters:
I20260926 20:54:39.905054 20160 leader_election.cc:288] T 00000000000000000000000000000000 P 054d424d4d37445a922616941cca4b36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:54:39.905103 20163 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 054d424d4d37445a922616941cca4b36 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:54:39.905263 20163 raft_consensus.cc:686] T 00000000000000000000000000000000 P 054d424d4d37445a922616941cca4b36 [term 1 LEADER]: Becoming Leader. State: Replica: 054d424d4d37445a922616941cca4b36, State: Running, Role: LEADER
I20260926 20:54:39.905299 20160 sys_catalog.cc:564] T 00000000000000000000000000000000 P 054d424d4d37445a922616941cca4b36 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:54:39.905360 20163 consensus_queue.cc:237] T 00000000000000000000000000000000 P 054d424d4d37445a922616941cca4b36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "054d424d4d37445a922616941cca4b36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39807 } }
I20260926 20:54:39.905768 20165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 054d424d4d37445a922616941cca4b36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 054d424d4d37445a922616941cca4b36. Latest consensus state: current_term: 1 leader_uuid: "054d424d4d37445a922616941cca4b36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "054d424d4d37445a922616941cca4b36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39807 } } }
I20260926 20:54:39.905867 20165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 054d424d4d37445a922616941cca4b36 [sys.catalog]: This master's current role is: LEADER
I20260926 20:54:39.905889 20164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 054d424d4d37445a922616941cca4b36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "054d424d4d37445a922616941cca4b36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "054d424d4d37445a922616941cca4b36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39807 } } }
I20260926 20:54:39.905939 20164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 054d424d4d37445a922616941cca4b36 [sys.catalog]: This master's current role is: LEADER
I20260926 20:54:39.906075 20172 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:54:39.906618 20172 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:54:39.907591 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 20092
I20260926 20:54:39.907706 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455725523895-392-0/minicluster-data/master-0/wal/instance
I20260926 20:54:39.907967 392 external_mini_cluster.cc:1428] Setting key 137fcc351f976b394cded9d0796ecdf2
I20260926 20:54:39.908257 20172 catalog_manager.cc:1347] Generated new cluster ID: c2741258f9cd443bafad065f805d8079
I20260926 20:54:39.908309 20172 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 20:54:39 dist-test-slave-hwlj krb5kdc[20079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 20:54:39.921016 20183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:39.908469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53019 (local address 127.0.0.1:39807)
0926 20:54:39.912115 (+ 3646us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:39.912123 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:39.912149 (+ 26us) server_negotiation.cc:407] Connection header received
0926 20:54:39.912254 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:39.912259 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:39.912316 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:39.912455 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:39.913089 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:39.914040 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:39.914802 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:39.915230 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:39.915344 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:39.917701 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:39.917705 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:39.917705 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:54:39.917716 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:39.919371 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:39.919781 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:39.919784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:39.919784 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:39.919879 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:39.920104 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:39.920105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:39.920105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:39.920392 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:39.920561 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:39.920674 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 20:54:39.920781 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":3605,"thread_start_us":74,"threads_started":1}
I20260926 20:54:39.922127 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:54:39.923386 20172 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:54:39.924108 20172 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:54:39.930578 20172 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 054d424d4d37445a922616941cca4b36: Generated new TSK 0
I20260926 20:54:39.930770 20172 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:54:39.931207 20183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:39.923673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58241 (local address 127.0.0.1:39807)
0926 20:54:39.923821 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:39.923823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:39.923831 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:54:39.923850 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:39.923851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:39.923859 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:39.923920 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:39.924589 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:39.925329 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:39.926355 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:39.926658 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:39.926717 (+ 59us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:39.928216 (+ 1499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:39.928217 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:39.928217 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:54:39.928224 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:39.929886 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:39.930373 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:39.930374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:39.930374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:39.930431 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:39.930673 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:39.930674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:39.930675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:39.930804 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:39.930904 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:39.931102 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 20:54:39.931153 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260926 20:54:39.933310 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 20092
W20260926 20:54:39.937736 20077 connection.cc:535] client connection to 127.0.0.1:39807 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39807): Connection reset by peer (error 104)
2026-09-26T20:54:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1322 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455725523895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:54:39 dist-test-slave-hwlj krb5kdc[20191](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:54:39 dist-test-slave-hwlj krb5kdc[20191](info): set up 2 sockets
Sep 26 20:54:39 dist-test-slave-hwlj krb5kdc[20191](info): commencing operation
krb5kdc: starting...
W20260926 20:54:40.967904 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455725523895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455725523895-392-0/krb5kdc/test-user.keytab.
Sep 26 20:54:41 dist-test-slave-hwlj krb5kdc[20191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:54:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:54:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455725523895-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455725523895-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:54:41.090847 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42243
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455725523895-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:54:41.171720 20204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:54:41.171984 20204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:54:41.172008 20204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:54:41.173095 20204 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:54:41.173132 20204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:54:41.173146 20204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:54:41.173159 20204 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:54:41.173171 20204 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:54:41.174578 20204 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455725523895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42243
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455725523895-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:54:41.174762 20204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:54:41.174929 20204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:54:41.179682 20210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:41.179698 20212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:41.179893 20209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:41.179915 20204 server_base.cc:1034] running on GCE node
I20260926 20:54:41.180145 20204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:54:41.180430 20204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:54:41.181579 20204 hybrid_clock.cc:648] HybridClock initialized: now 1790456081181559 us; error 38 us; skew 500 ppm
Sep 26 20:54:41 dist-test-slave-hwlj krb5kdc[20191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456081, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:54:41.190493 20204 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:54:41.190915 20204 webserver.cc:460] Webserver started at http://127.0.0.1:41757/ using document root <none> and password file <none>
I20260926 20:54:41.191133 20204 fs_manager.cc:359] Metadata directory not provided
I20260926 20:54:41.191174 20204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:54:41.191272 20204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:54:41.192373 20204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455725523895-392-0/minicluster-data/master-0/data/instance:
uuid: "0d07626f2f884f7eb4d02b9b4e7052e2"
format_stamp: "Formatted at 2026-09-26 20:54:41 on dist-test-slave-hwlj"
server_key: "b7cbb6c70871dfe7a4bc7b1227cc9b83"
server_key_iv: "1a5a621a3e3de8e468cbbc7af601760f"
server_key_version: "encryptionkey@0"
I20260926 20:54:41.192723 20204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455725523895-392-0/minicluster-data/master-0/wal/instance:
uuid: "0d07626f2f884f7eb4d02b9b4e7052e2"
format_stamp: "Formatted at 2026-09-26 20:54:41 on dist-test-slave-hwlj"
server_key: "b7cbb6c70871dfe7a4bc7b1227cc9b83"
server_key_iv: "1a5a621a3e3de8e468cbbc7af601760f"
server_key_version: "encryptionkey@0"
I20260926 20:54:41.193986 20204 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:54:41.194657 20219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:41.194866 20204 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 20:54:41.194950 20204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "0d07626f2f884f7eb4d02b9b4e7052e2"
format_stamp: "Formatted at 2026-09-26 20:54:41 on dist-test-slave-hwlj"
server_key: "b7cbb6c70871dfe7a4bc7b1227cc9b83"
server_key_iv: "1a5a621a3e3de8e468cbbc7af601760f"
server_key_version: "encryptionkey@0"
I20260926 20:54:41.195004 20204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:54:41.209589 20204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:54:41.211912 20204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:54:41.212029 20204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:54:41.216090 20204 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42243
I20260926 20:54:41.216125 20271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42243 every 8 connection(s)
I20260926 20:54:41.216490 20204 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:54:41.217373 20272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:54:41.219626 20272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d07626f2f884f7eb4d02b9b4e7052e2: Bootstrap starting.
I20260926 20:54:41.220263 20272 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d07626f2f884f7eb4d02b9b4e7052e2: Neither blocks nor log segments found. Creating new log.
I20260926 20:54:41.220498 20272 log.cc:824] T 00000000000000000000000000000000 P 0d07626f2f884f7eb4d02b9b4e7052e2: Log is configured to *not* fsync() on all Append() calls
I20260926 20:54:41.221264 20272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d07626f2f884f7eb4d02b9b4e7052e2: No bootstrap required, opened a new log
I20260926 20:54:41.222847 20272 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d07626f2f884f7eb4d02b9b4e7052e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d07626f2f884f7eb4d02b9b4e7052e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42243 } }
I20260926 20:54:41.223003 20272 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d07626f2f884f7eb4d02b9b4e7052e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:54:41.223026 20272 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d07626f2f884f7eb4d02b9b4e7052e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d07626f2f884f7eb4d02b9b4e7052e2, State: Initialized, Role: FOLLOWER
I20260926 20:54:41.223187 20272 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d07626f2f884f7eb4d02b9b4e7052e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d07626f2f884f7eb4d02b9b4e7052e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42243 } }
I20260926 20:54:41.223277 20272 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d07626f2f884f7eb4d02b9b4e7052e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:54:41.223301 20272 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d07626f2f884f7eb4d02b9b4e7052e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:54:41.223330 20272 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d07626f2f884f7eb4d02b9b4e7052e2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:54:41.223922 20272 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d07626f2f884f7eb4d02b9b4e7052e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d07626f2f884f7eb4d02b9b4e7052e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42243 } }
I20260926 20:54:41.224012 20272 leader_election.cc:302] T 00000000000000000000000000000000 P 0d07626f2f884f7eb4d02b9b4e7052e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d07626f2f884f7eb4d02b9b4e7052e2; no voters:
I20260926 20:54:41.224155 20272 leader_election.cc:288] T 00000000000000000000000000000000 P 0d07626f2f884f7eb4d02b9b4e7052e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:54:41.224224 20275 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d07626f2f884f7eb4d02b9b4e7052e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:54:41.224380 20275 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d07626f2f884f7eb4d02b9b4e7052e2 [term 1 LEADER]: Becoming Leader. State: Replica: 0d07626f2f884f7eb4d02b9b4e7052e2, State: Running, Role: LEADER
I20260926 20:54:41.224443 20272 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d07626f2f884f7eb4d02b9b4e7052e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:54:41.224473 20275 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d07626f2f884f7eb4d02b9b4e7052e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d07626f2f884f7eb4d02b9b4e7052e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42243 } }
I20260926 20:54:41.224893 20277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d07626f2f884f7eb4d02b9b4e7052e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d07626f2f884f7eb4d02b9b4e7052e2. Latest consensus state: current_term: 1 leader_uuid: "0d07626f2f884f7eb4d02b9b4e7052e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d07626f2f884f7eb4d02b9b4e7052e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42243 } } }
I20260926 20:54:41.224985 20277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d07626f2f884f7eb4d02b9b4e7052e2 [sys.catalog]: This master's current role is: LEADER
I20260926 20:54:41.225028 20276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d07626f2f884f7eb4d02b9b4e7052e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d07626f2f884f7eb4d02b9b4e7052e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d07626f2f884f7eb4d02b9b4e7052e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42243 } } }
I20260926 20:54:41.225096 20276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d07626f2f884f7eb4d02b9b4e7052e2 [sys.catalog]: This master's current role is: LEADER
I20260926 20:54:41.225225 20282 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:54:41.225746 20282 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:54:41.225736 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 20204
I20260926 20:54:41.225854 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455725523895-392-0/minicluster-data/master-0/wal/instance
I20260926 20:54:41.226075 392 external_mini_cluster.cc:1428] Setting key 9de19ced225bf5cd8e9651380de6b1a9
I20260926 20:54:41.227830 20282 catalog_manager.cc:1347] Generated new cluster ID: 4ed4399a942842c6a7fa04e62accf657
I20260926 20:54:41.227890 20282 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:54:41.232318 20294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:41.226572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54329 (local address 127.0.0.1:42243)
0926 20:54:41.226764 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:41.226775 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:41.226811 (+ 36us) server_negotiation.cc:407] Connection header received
0926 20:54:41.226939 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:41.226946 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:41.227001 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:41.227147 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:41.227539 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:41.228963 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:41.230120 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:41.230753 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:41.230861 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:41.231589 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:41.231592 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:41.231593 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:54:41.231830 (+ 237us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:41.231868 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:54:41.231883 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:41.231961 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:41.232075 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 20:54:41.232100 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":67,"threads_started":1}
I20260926 20:54:41.233187 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:54:41.238337 20294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:41.234752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37641 (local address 127.0.0.1:42243)
0926 20:54:41.234784 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:41.234786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:41.234946 (+ 160us) server_negotiation.cc:407] Connection header received
0926 20:54:41.235010 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:41.235011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:41.235020 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:41.235067 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:41.235214 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:41.236020 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:41.237029 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:41.237328 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:41.237386 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:41.238088 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:41.238089 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:41.238090 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:54:41.238094 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:41.238109 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:54:41.238113 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:41.238163 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:41.238252 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 20:54:41.238264 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260926 20:54:41.241560 20282 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:54:41.242275 20282 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:54:41.254858 20282 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d07626f2f884f7eb4d02b9b4e7052e2: Generated new TSK 0
I20260926 20:54:41.255092 20282 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:54:41.261060 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 20204
W20260926 20:54:41.265359 20189 connection.cc:535] client connection to 127.0.0.1:42243 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42243): Connection reset by peer (error 104)
2026-09-26T20:54:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1328 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455725523895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:54:41 dist-test-slave-hwlj krb5kdc[20303](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:54:41 dist-test-slave-hwlj krb5kdc[20303](info): set up 2 sockets
Sep 26 20:54:41 dist-test-slave-hwlj krb5kdc[20303](info): commencing operation
krb5kdc: starting...
W20260926 20:54:42.291671 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455725523895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455725523895-392-0/krb5kdc/test-user.keytab.
Sep 26 20:54:42 dist-test-slave-hwlj krb5kdc[20303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:54:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:54:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455725523895-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455725523895-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:54:42.410527 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36627
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455725523895-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36627
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:54:42.496948 20316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:54:42.497212 20316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:54:42.497262 20316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:54:42.498703 20316 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:54:42.498791 20316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:54:42.498821 20316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:54:42.498853 20316 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:54:42.498878 20316 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:54:42.500509 20316 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455725523895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36627
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455725523895-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:54:42.500820 20316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:54:42.501065 20316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:54:42.506191 20324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:42.506332 20321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:42.506376 20316 server_base.cc:1034] running on GCE node
W20260926 20:54:42.506419 20322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:42.506815 20316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:54:42.507169 20316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:54:42.508322 20316 hybrid_clock.cc:648] HybridClock initialized: now 1790456082508303 us; error 44 us; skew 500 ppm
Sep 26 20:54:42 dist-test-slave-hwlj krb5kdc[20303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:54:42.510749 20316 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:54:42.511209 20316 webserver.cc:460] Webserver started at http://127.0.0.1:45579/ using document root <none> and password file <none>
I20260926 20:54:42.511412 20316 fs_manager.cc:359] Metadata directory not provided
I20260926 20:54:42.511452 20316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:54:42.511541 20316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:54:42.512631 20316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455725523895-392-0/minicluster-data/master-0/data/instance:
uuid: "da8d0496320c4fbd90e82876a3abe179"
format_stamp: "Formatted at 2026-09-26 20:54:42 on dist-test-slave-hwlj"
server_key: "fe5b77a023ad5d5cfa59cb9c9b2981e4"
server_key_iv: "86c0a6623025df8b862c62d6fcc855d5"
server_key_version: "encryptionkey@0"
I20260926 20:54:42.512933 20316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455725523895-392-0/minicluster-data/master-0/wal/instance:
uuid: "da8d0496320c4fbd90e82876a3abe179"
format_stamp: "Formatted at 2026-09-26 20:54:42 on dist-test-slave-hwlj"
server_key: "fe5b77a023ad5d5cfa59cb9c9b2981e4"
server_key_iv: "86c0a6623025df8b862c62d6fcc855d5"
server_key_version: "encryptionkey@0"
I20260926 20:54:42.514180 20316 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 20:54:42.514971 20331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:42.515183 20316 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:54:42.515296 20316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "da8d0496320c4fbd90e82876a3abe179"
format_stamp: "Formatted at 2026-09-26 20:54:42 on dist-test-slave-hwlj"
server_key: "fe5b77a023ad5d5cfa59cb9c9b2981e4"
server_key_iv: "86c0a6623025df8b862c62d6fcc855d5"
server_key_version: "encryptionkey@0"
I20260926 20:54:42.515373 20316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:54:42.518643 20316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:54:42.521193 20316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:54:42.521309 20316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:54:42.525162 20316 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36627
I20260926 20:54:42.525192 20383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36627 every 8 connection(s)
I20260926 20:54:42.525552 20316 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:54:42.526567 20384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:54:42.528995 20384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da8d0496320c4fbd90e82876a3abe179: Bootstrap starting.
I20260926 20:54:42.529610 20384 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da8d0496320c4fbd90e82876a3abe179: Neither blocks nor log segments found. Creating new log.
I20260926 20:54:42.530035 20384 log.cc:824] T 00000000000000000000000000000000 P da8d0496320c4fbd90e82876a3abe179: Log is configured to *not* fsync() on all Append() calls
I20260926 20:54:42.530853 20384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da8d0496320c4fbd90e82876a3abe179: No bootstrap required, opened a new log
I20260926 20:54:42.532634 20384 raft_consensus.cc:348] T 00000000000000000000000000000000 P da8d0496320c4fbd90e82876a3abe179 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8d0496320c4fbd90e82876a3abe179" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36627 } }
I20260926 20:54:42.532788 20384 raft_consensus.cc:374] T 00000000000000000000000000000000 P da8d0496320c4fbd90e82876a3abe179 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:54:42.532811 20384 raft_consensus.cc:729] T 00000000000000000000000000000000 P da8d0496320c4fbd90e82876a3abe179 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da8d0496320c4fbd90e82876a3abe179, State: Initialized, Role: FOLLOWER
I20260926 20:54:42.532974 20384 consensus_queue.cc:260] T 00000000000000000000000000000000 P da8d0496320c4fbd90e82876a3abe179 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8d0496320c4fbd90e82876a3abe179" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36627 } }
I20260926 20:54:42.533064 20384 raft_consensus.cc:388] T 00000000000000000000000000000000 P da8d0496320c4fbd90e82876a3abe179 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:54:42.533089 20384 raft_consensus.cc:482] T 00000000000000000000000000000000 P da8d0496320c4fbd90e82876a3abe179 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:54:42.533156 20384 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da8d0496320c4fbd90e82876a3abe179 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:54:42.533774 20384 raft_consensus.cc:504] T 00000000000000000000000000000000 P da8d0496320c4fbd90e82876a3abe179 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8d0496320c4fbd90e82876a3abe179" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36627 } }
I20260926 20:54:42.533864 20384 leader_election.cc:302] T 00000000000000000000000000000000 P da8d0496320c4fbd90e82876a3abe179 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da8d0496320c4fbd90e82876a3abe179; no voters:
I20260926 20:54:42.534093 20384 leader_election.cc:288] T 00000000000000000000000000000000 P da8d0496320c4fbd90e82876a3abe179 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:54:42.534144 20387 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da8d0496320c4fbd90e82876a3abe179 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:54:42.534379 20384 sys_catalog.cc:564] T 00000000000000000000000000000000 P da8d0496320c4fbd90e82876a3abe179 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:54:42.534590 20387 raft_consensus.cc:686] T 00000000000000000000000000000000 P da8d0496320c4fbd90e82876a3abe179 [term 1 LEADER]: Becoming Leader. State: Replica: da8d0496320c4fbd90e82876a3abe179, State: Running, Role: LEADER
I20260926 20:54:42.534701 20387 consensus_queue.cc:237] T 00000000000000000000000000000000 P da8d0496320c4fbd90e82876a3abe179 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8d0496320c4fbd90e82876a3abe179" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36627 } }
I20260926 20:54:42.535120 20389 sys_catalog.cc:455] T 00000000000000000000000000000000 P da8d0496320c4fbd90e82876a3abe179 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da8d0496320c4fbd90e82876a3abe179. Latest consensus state: current_term: 1 leader_uuid: "da8d0496320c4fbd90e82876a3abe179" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8d0496320c4fbd90e82876a3abe179" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36627 } } }
I20260926 20:54:42.535224 20389 sys_catalog.cc:458] T 00000000000000000000000000000000 P da8d0496320c4fbd90e82876a3abe179 [sys.catalog]: This master's current role is: LEADER
I20260926 20:54:42.535463 20396 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:54:42.535593 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 20316
I20260926 20:54:42.535712 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455725523895-392-0/minicluster-data/master-0/wal/instance
I20260926 20:54:42.535984 392 external_mini_cluster.cc:1428] Setting key d4715d8a09877776d073e1b6b103abce
I20260926 20:54:42.535110 20388 sys_catalog.cc:455] T 00000000000000000000000000000000 P da8d0496320c4fbd90e82876a3abe179 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da8d0496320c4fbd90e82876a3abe179" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8d0496320c4fbd90e82876a3abe179" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36627 } } }
I20260926 20:54:42.536136 20396 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:54:42.536157 20388 sys_catalog.cc:458] T 00000000000000000000000000000000 P da8d0496320c4fbd90e82876a3abe179 [sys.catalog]: This master's current role is: LEADER
I20260926 20:54:42.538131 20396 catalog_manager.cc:1347] Generated new cluster ID: f1d1c2c8328a4523b7e26e61eefcff86
I20260926 20:54:42.538188 20396 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:54:42.538548 20406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:42.536635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33367 (local address 127.0.0.1:36627)
0926 20:54:42.536897 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:42.536907 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:42.536937 (+ 30us) server_negotiation.cc:407] Connection header received
0926 20:54:42.537058 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:42.537077 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:42.537110 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:42.537362 (+ 252us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:42.538001 (+ 639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:42.538005 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:42.538005 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:54:42.538009 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:42.538050 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:54:42.538081 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:42.538139 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:42.538328 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 20:54:42.538370 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":86,"threads_started":1}
I20260926 20:54:42.539413 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:54:42.542079 20406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:42.541075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38121 (local address 127.0.0.1:36627)
0926 20:54:42.541109 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:42.541111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:42.541118 (+ 7us) server_negotiation.cc:407] Connection header received
0926 20:54:42.541139 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:42.541140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:42.541156 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:42.541216 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:42.541835 (+ 619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:42.541836 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:42.541837 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:54:42.541841 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:42.541857 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:54:42.541862 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:42.541927 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:42.541980 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 20:54:42.542009 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 20:54:42.552232 20396 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:54:42.552474 20396 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:54:42.561929 20396 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da8d0496320c4fbd90e82876a3abe179: Generated new TSK 0
I20260926 20:54:42.562148 20396 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:54:42.567480 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 20316
2026-09-26T20:54:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1306 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455725523895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:54:42 dist-test-slave-hwlj krb5kdc[20415](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:54:42 dist-test-slave-hwlj krb5kdc[20415](info): set up 2 sockets
Sep 26 20:54:42 dist-test-slave-hwlj krb5kdc[20415](info): commencing operation
krb5kdc: starting...
W20260926 20:54:43.603888 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455725523895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455725523895-392-0/krb5kdc/test-user.keytab.
Sep 26 20:54:43 dist-test-slave-hwlj krb5kdc[20415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:54:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:54:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455725523895-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455725523895-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:54:43.728091 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38885
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455725523895-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:54:43.812956 20428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:54:43.813161 20428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:54:43.813185 20428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:54:43.814265 20428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:54:43.814298 20428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:54:43.814312 20428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:54:43.814324 20428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:54:43.814337 20428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:54:43.815727 20428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455725523895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455725523895-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:54:43.815934 20428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:54:43.816094 20428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:54:43.821188 20433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:43.821211 20434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:43.821198 20436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:43.821574 20428 server_base.cc:1034] running on GCE node
I20260926 20:54:43.821883 20428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:54:43.822224 20428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:54:43.823386 20428 hybrid_clock.cc:648] HybridClock initialized: now 1790456083823370 us; error 27 us; skew 500 ppm
Sep 26 20:54:43 dist-test-slave-hwlj krb5kdc[20415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:54:43.826079 20428 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:54:43.826542 20428 webserver.cc:460] Webserver started at http://127.0.0.1:43117/ using document root <none> and password file <none>
I20260926 20:54:43.826771 20428 fs_manager.cc:359] Metadata directory not provided
I20260926 20:54:43.826835 20428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:54:43.826979 20428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:54:43.828154 20428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455725523895-392-0/minicluster-data/master-0/data/instance:
uuid: "dfd46cac304e4250891728cb80fb4e37"
format_stamp: "Formatted at 2026-09-26 20:54:43 on dist-test-slave-hwlj"
server_key: "e79a314d9cab98fbcdff52ebf2ff29f8"
server_key_iv: "26f2057bbd6ede077632cca77047594f"
server_key_version: "encryptionkey@0"
I20260926 20:54:43.828518 20428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455725523895-392-0/minicluster-data/master-0/wal/instance:
uuid: "dfd46cac304e4250891728cb80fb4e37"
format_stamp: "Formatted at 2026-09-26 20:54:43 on dist-test-slave-hwlj"
server_key: "e79a314d9cab98fbcdff52ebf2ff29f8"
server_key_iv: "26f2057bbd6ede077632cca77047594f"
server_key_version: "encryptionkey@0"
I20260926 20:54:43.829879 20428 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:54:43.830716 20443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:43.830907 20428 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:54:43.831010 20428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "dfd46cac304e4250891728cb80fb4e37"
format_stamp: "Formatted at 2026-09-26 20:54:43 on dist-test-slave-hwlj"
server_key: "e79a314d9cab98fbcdff52ebf2ff29f8"
server_key_iv: "26f2057bbd6ede077632cca77047594f"
server_key_version: "encryptionkey@0"
I20260926 20:54:43.831086 20428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:54:43.839856 20428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:54:43.841673 20428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:54:43.841831 20428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:54:43.845983 20428 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38885
I20260926 20:54:43.846026 20495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38885 every 8 connection(s)
I20260926 20:54:43.846390 20428 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:54:43.847222 20496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:54:43.849490 20496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfd46cac304e4250891728cb80fb4e37: Bootstrap starting.
I20260926 20:54:43.850044 20496 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfd46cac304e4250891728cb80fb4e37: Neither blocks nor log segments found. Creating new log.
I20260926 20:54:43.850324 20496 log.cc:824] T 00000000000000000000000000000000 P dfd46cac304e4250891728cb80fb4e37: Log is configured to *not* fsync() on all Append() calls
I20260926 20:54:43.851044 20496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfd46cac304e4250891728cb80fb4e37: No bootstrap required, opened a new log
I20260926 20:54:43.852677 20496 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfd46cac304e4250891728cb80fb4e37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd46cac304e4250891728cb80fb4e37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38885 } }
I20260926 20:54:43.852810 20496 raft_consensus.cc:374] T 00000000000000000000000000000000 P dfd46cac304e4250891728cb80fb4e37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:54:43.852833 20496 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfd46cac304e4250891728cb80fb4e37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfd46cac304e4250891728cb80fb4e37, State: Initialized, Role: FOLLOWER
I20260926 20:54:43.852998 20496 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfd46cac304e4250891728cb80fb4e37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd46cac304e4250891728cb80fb4e37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38885 } }
I20260926 20:54:43.853091 20496 raft_consensus.cc:388] T 00000000000000000000000000000000 P dfd46cac304e4250891728cb80fb4e37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:54:43.853135 20496 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfd46cac304e4250891728cb80fb4e37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:54:43.853134 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 20428
I20260926 20:54:43.853192 20496 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfd46cac304e4250891728cb80fb4e37 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:54:43.853231 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455725523895-392-0/minicluster-data/master-0/wal/instance
I20260926 20:54:43.853425 392 external_mini_cluster.cc:1428] Setting key cdb01b67b681b2d1e7d578c1d8d503d2
I20260926 20:54:43.853895 20496 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfd46cac304e4250891728cb80fb4e37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd46cac304e4250891728cb80fb4e37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38885 } }
I20260926 20:54:43.854007 20496 leader_election.cc:302] T 00000000000000000000000000000000 P dfd46cac304e4250891728cb80fb4e37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dfd46cac304e4250891728cb80fb4e37; no voters:
I20260926 20:54:43.854180 20496 leader_election.cc:288] T 00000000000000000000000000000000 P dfd46cac304e4250891728cb80fb4e37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:54:43.854224 20501 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfd46cac304e4250891728cb80fb4e37 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:54:43.854404 20501 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfd46cac304e4250891728cb80fb4e37 [term 1 LEADER]: Becoming Leader. State: Replica: dfd46cac304e4250891728cb80fb4e37, State: Running, Role: LEADER
I20260926 20:54:43.854465 20496 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfd46cac304e4250891728cb80fb4e37 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:54:43.854540 20501 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfd46cac304e4250891728cb80fb4e37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd46cac304e4250891728cb80fb4e37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38885 } }
I20260926 20:54:43.855166 20502 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfd46cac304e4250891728cb80fb4e37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfd46cac304e4250891728cb80fb4e37. Latest consensus state: current_term: 1 leader_uuid: "dfd46cac304e4250891728cb80fb4e37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd46cac304e4250891728cb80fb4e37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38885 } } }
I20260926 20:54:43.855265 20502 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfd46cac304e4250891728cb80fb4e37 [sys.catalog]: This master's current role is: LEADER
I20260926 20:54:43.855573 20503 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfd46cac304e4250891728cb80fb4e37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dfd46cac304e4250891728cb80fb4e37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd46cac304e4250891728cb80fb4e37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38885 } } }
I20260926 20:54:43.855772 20503 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfd46cac304e4250891728cb80fb4e37 [sys.catalog]: This master's current role is: LEADER
I20260926 20:54:43.855906 20510 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:54:43.856518 20510 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:54:43.857965 20510 catalog_manager.cc:1347] Generated new cluster ID: 43fb51e7185f40a484ea3ca608843107
I20260926 20:54:43.858017 20510 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 20:54:43 dist-test-slave-hwlj krb5kdc[20415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 20:54:43.865451 20500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:43.853888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51281 (local address 127.0.0.1:38885)
0926 20:54:43.854109 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:43.854118 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:43.854156 (+ 38us) server_negotiation.cc:407] Connection header received
0926 20:54:43.854312 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:43.854320 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:43.854386 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:43.854534 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:43.855402 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:43.856840 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:43.857923 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:43.858519 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:43.858636 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:43.861342 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:43.861346 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:43.861346 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:54:43.861359 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:43.863419 (+ 2060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:43.863871 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:43.863875 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:43.863875 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:43.863995 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:43.864209 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:43.864210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:43.864210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:43.864503 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:43.864687 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:43.865119 (+ 432us) server_negotiation.cc:299] Negotiation successful
0926 20:54:43.865256 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":75,"threads_started":1}
I20260926 20:54:43.866489 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:54:43.875250 20500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:43.868022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55143 (local address 127.0.0.1:38885)
0926 20:54:43.868064 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:43.868065 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:43.868133 (+ 68us) server_negotiation.cc:407] Connection header received
0926 20:54:43.868227 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:43.868228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:43.868237 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:43.868265 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:43.868961 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:43.869711 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:43.870766 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:43.871071 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:43.871110 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:43.872565 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:43.872566 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:43.872566 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:54:43.872573 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:43.874070 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:43.874445 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:43.874446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:43.874446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:43.874494 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:43.874689 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:43.874689 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:43.874690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:43.874807 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:43.874874 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:43.875083 (+ 209us) server_negotiation.cc:299] Negotiation successful
0926 20:54:43.875159 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 20:54:43.885499 20510 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:54:43.886230 20510 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:54:43.892657 20510 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dfd46cac304e4250891728cb80fb4e37: Generated new TSK 0
I20260926 20:54:43.892856 20510 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:54:43.899976 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 20428
2026-09-26T20:54:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1331 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455725523895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:54:43 dist-test-slave-hwlj krb5kdc[20527](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:54:43 dist-test-slave-hwlj krb5kdc[20527](info): set up 2 sockets
Sep 26 20:54:43 dist-test-slave-hwlj krb5kdc[20527](info): commencing operation
krb5kdc: starting...
W20260926 20:54:44.927623 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455725523895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455725523895-392-0/krb5kdc/test-user.keytab.
Sep 26 20:54:45 dist-test-slave-hwlj krb5kdc[20527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:54:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:54:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455725523895-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455725523895-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:54:45.047310 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34689
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455725523895-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:54:45.127895 20540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:54:45.128160 20540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:54:45.128201 20540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:54:45.129398 20540 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:54:45.129482 20540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:54:45.129504 20540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:54:45.129524 20540 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:54:45.129554 20540 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:54:45.131099 20540 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455725523895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455725523895-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:54:45.131399 20540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:54:45.131642 20540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:54:45.136583 20545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:45.136583 20548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:45.136631 20546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:45.136823 20540 server_base.cc:1034] running on GCE node
I20260926 20:54:45.137095 20540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:54:45.137411 20540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:54:45.138605 20540 hybrid_clock.cc:648] HybridClock initialized: now 1790456085138579 us; error 41 us; skew 500 ppm
Sep 26 20:54:45 dist-test-slave-hwlj krb5kdc[20527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:54:45.141296 20540 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:54:45.141803 20540 webserver.cc:460] Webserver started at http://127.0.0.1:38155/ using document root <none> and password file <none>
I20260926 20:54:45.142047 20540 fs_manager.cc:359] Metadata directory not provided
I20260926 20:54:45.142117 20540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:54:45.142267 20540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:54:45.143369 20540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455725523895-392-0/minicluster-data/master-0/data/instance:
uuid: "a29bcd3cd55f4208a5046ad60214d951"
format_stamp: "Formatted at 2026-09-26 20:54:45 on dist-test-slave-hwlj"
server_key: "394b2d01649601e22ea9064fc4a38a37"
server_key_iv: "074cda69ec31169416f030ed32685955"
server_key_version: "encryptionkey@0"
I20260926 20:54:45.143843 20540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455725523895-392-0/minicluster-data/master-0/wal/instance:
uuid: "a29bcd3cd55f4208a5046ad60214d951"
format_stamp: "Formatted at 2026-09-26 20:54:45 on dist-test-slave-hwlj"
server_key: "394b2d01649601e22ea9064fc4a38a37"
server_key_iv: "074cda69ec31169416f030ed32685955"
server_key_version: "encryptionkey@0"
I20260926 20:54:45.145311 20540 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:54:45.146188 20555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:45.146376 20540 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:45.146471 20540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "a29bcd3cd55f4208a5046ad60214d951"
format_stamp: "Formatted at 2026-09-26 20:54:45 on dist-test-slave-hwlj"
server_key: "394b2d01649601e22ea9064fc4a38a37"
server_key_iv: "074cda69ec31169416f030ed32685955"
server_key_version: "encryptionkey@0"
I20260926 20:54:45.146528 20540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:54:45.181031 20540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:54:45.183494 20540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:54:45.183619 20540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:54:45.187561 20540 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34689
I20260926 20:54:45.187619 20607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34689 every 8 connection(s)
I20260926 20:54:45.188009 20540 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:54:45.188997 20608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:54:45.191269 20608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a29bcd3cd55f4208a5046ad60214d951: Bootstrap starting.
I20260926 20:54:45.191871 20608 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a29bcd3cd55f4208a5046ad60214d951: Neither blocks nor log segments found. Creating new log.
I20260926 20:54:45.192211 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 20540
I20260926 20:54:45.192263 20608 log.cc:824] T 00000000000000000000000000000000 P a29bcd3cd55f4208a5046ad60214d951: Log is configured to *not* fsync() on all Append() calls
I20260926 20:54:45.192332 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455725523895-392-0/minicluster-data/master-0/wal/instance
I20260926 20:54:45.192539 392 external_mini_cluster.cc:1428] Setting key 1361072b4ebc2bc804832c65ee89a01d
I20260926 20:54:45.193173 20608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a29bcd3cd55f4208a5046ad60214d951: No bootstrap required, opened a new log
I20260926 20:54:45.195735 20608 raft_consensus.cc:348] T 00000000000000000000000000000000 P a29bcd3cd55f4208a5046ad60214d951 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a29bcd3cd55f4208a5046ad60214d951" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34689 } }
I20260926 20:54:45.195940 20608 raft_consensus.cc:374] T 00000000000000000000000000000000 P a29bcd3cd55f4208a5046ad60214d951 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:54:45.195971 20608 raft_consensus.cc:729] T 00000000000000000000000000000000 P a29bcd3cd55f4208a5046ad60214d951 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29bcd3cd55f4208a5046ad60214d951, State: Initialized, Role: FOLLOWER
I20260926 20:54:45.196096 20608 consensus_queue.cc:260] T 00000000000000000000000000000000 P a29bcd3cd55f4208a5046ad60214d951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a29bcd3cd55f4208a5046ad60214d951" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34689 } }
I20260926 20:54:45.196182 20608 raft_consensus.cc:388] T 00000000000000000000000000000000 P a29bcd3cd55f4208a5046ad60214d951 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:54:45.196273 20608 raft_consensus.cc:482] T 00000000000000000000000000000000 P a29bcd3cd55f4208a5046ad60214d951 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:54:45.196316 20608 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a29bcd3cd55f4208a5046ad60214d951 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:54:45.197012 20608 raft_consensus.cc:504] T 00000000000000000000000000000000 P a29bcd3cd55f4208a5046ad60214d951 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a29bcd3cd55f4208a5046ad60214d951" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34689 } }
I20260926 20:54:45.197170 20608 leader_election.cc:302] T 00000000000000000000000000000000 P a29bcd3cd55f4208a5046ad60214d951 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a29bcd3cd55f4208a5046ad60214d951; no voters:
I20260926 20:54:45.197371 20608 leader_election.cc:288] T 00000000000000000000000000000000 P a29bcd3cd55f4208a5046ad60214d951 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:54:45.197510 20613 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a29bcd3cd55f4208a5046ad60214d951 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:54:45.197686 20613 raft_consensus.cc:686] T 00000000000000000000000000000000 P a29bcd3cd55f4208a5046ad60214d951 [term 1 LEADER]: Becoming Leader. State: Replica: a29bcd3cd55f4208a5046ad60214d951, State: Running, Role: LEADER
I20260926 20:54:45.197692 20608 sys_catalog.cc:564] T 00000000000000000000000000000000 P a29bcd3cd55f4208a5046ad60214d951 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:54:45.198382 20612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:45.193092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55689 (local address 127.0.0.1:34689)
0926 20:54:45.193271 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:45.193281 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:45.193315 (+ 34us) server_negotiation.cc:407] Connection header received
0926 20:54:45.193732 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:45.193738 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:45.193787 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:45.193900 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:45.194143 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:45.195008 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:45.196220 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:45.196723 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:45.196813 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:45.197618 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:45.197622 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:45.197622 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:54:45.197836 (+ 214us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:45.197882 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:54:45.197917 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:45.197955 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:45.198088 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 20:54:45.198168 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":91,"threads_started":1}
I20260926 20:54:45.199185 20613 consensus_queue.cc:237] T 00000000000000000000000000000000 P a29bcd3cd55f4208a5046ad60214d951 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a29bcd3cd55f4208a5046ad60214d951" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34689 } }
I20260926 20:54:45.199359 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:54:45.199958 20614 sys_catalog.cc:455] T 00000000000000000000000000000000 P a29bcd3cd55f4208a5046ad60214d951 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a29bcd3cd55f4208a5046ad60214d951" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a29bcd3cd55f4208a5046ad60214d951" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34689 } } }
I20260926 20:54:45.200043 20614 sys_catalog.cc:458] T 00000000000000000000000000000000 P a29bcd3cd55f4208a5046ad60214d951 [sys.catalog]: This master's current role is: LEADER
I20260926 20:54:45.200268 20615 sys_catalog.cc:455] T 00000000000000000000000000000000 P a29bcd3cd55f4208a5046ad60214d951 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a29bcd3cd55f4208a5046ad60214d951. Latest consensus state: current_term: 1 leader_uuid: "a29bcd3cd55f4208a5046ad60214d951" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a29bcd3cd55f4208a5046ad60214d951" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34689 } } }
W20260926 20:54:45.200271 20627 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a29bcd3cd55f4208a5046ad60214d951: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260926 20:54:45.200414 20615 sys_catalog.cc:458] T 00000000000000000000000000000000 P a29bcd3cd55f4208a5046ad60214d951 [sys.catalog]: This master's current role is: LEADER
W20260926 20:54:45.200453 20627 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:54:45.200575 20629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:54:45.200773 20629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:54:45.202188 20629 catalog_manager.cc:1347] Generated new cluster ID: f6f46b052c6d45fab0c04e18a88fa07d
I20260926 20:54:45.202230 20629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:54:45.215610 20612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:45.204147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42153 (local address 127.0.0.1:34689)
0926 20:54:45.204196 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:45.204198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:45.212013 (+ 7815us) server_negotiation.cc:407] Connection header received
0926 20:54:45.212090 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:45.212091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:45.212106 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:45.212177 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:45.212365 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:45.213112 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:45.214192 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:45.214494 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:45.214558 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:45.215341 (+ 783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:45.215342 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:45.215343 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:54:45.215348 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:45.215364 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:54:45.215369 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:45.215419 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:45.215496 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 20:54:45.215520 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 20:54:45.226468 20629 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:54:45.227110 20629 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:54:45.234723 20629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a29bcd3cd55f4208a5046ad60214d951: Generated new TSK 0
I20260926 20:54:45.234948 20629 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:54:45.238556 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 20540
2026-09-26T20:54:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1338 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455725523895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:54:45 dist-test-slave-hwlj krb5kdc[20639](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:54:45 dist-test-slave-hwlj krb5kdc[20639](info): set up 2 sockets
Sep 26 20:54:45 dist-test-slave-hwlj krb5kdc[20639](info): commencing operation
krb5kdc: starting...
W20260926 20:54:46.272291 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455725523895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455725523895-392-0/krb5kdc/test-user.keytab.
Sep 26 20:54:46 dist-test-slave-hwlj krb5kdc[20639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:54:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:54:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455725523895-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455725523895-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:54:46.395401 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40351
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455725523895-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:54:46.475934 20652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:54:46.476198 20652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:54:46.476239 20652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:54:46.477443 20652 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:54:46.477519 20652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:54:46.477543 20652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:54:46.477574 20652 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:54:46.477605 20652 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:54:46.479105 20652 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455725523895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455725523895-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:54:46.479388 20652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:54:46.479620 20652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:54:46.484566 20660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:46.484486 20657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:46.484730 20652 server_base.cc:1034] running on GCE node
W20260926 20:54:46.484483 20658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:46.485071 20652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:54:46.485397 20652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:54:46.486562 20652 hybrid_clock.cc:648] HybridClock initialized: now 1790456086486553 us; error 39 us; skew 500 ppm
Sep 26 20:54:46 dist-test-slave-hwlj krb5kdc[20639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:54:46.495278 20652 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:54:46.495837 20652 webserver.cc:460] Webserver started at http://127.0.0.1:43971/ using document root <none> and password file <none>
I20260926 20:54:46.496031 20652 fs_manager.cc:359] Metadata directory not provided
I20260926 20:54:46.496073 20652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:54:46.496163 20652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:54:46.497205 20652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455725523895-392-0/minicluster-data/master-0/data/instance:
uuid: "2d5c640289a64c8a9aa1208376197f55"
format_stamp: "Formatted at 2026-09-26 20:54:46 on dist-test-slave-hwlj"
server_key: "200c75251ab814236e6b23d7bc406306"
server_key_iv: "9574277f9ddf351079714ff13145796c"
server_key_version: "encryptionkey@0"
I20260926 20:54:46.497511 20652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455725523895-392-0/minicluster-data/master-0/wal/instance:
uuid: "2d5c640289a64c8a9aa1208376197f55"
format_stamp: "Formatted at 2026-09-26 20:54:46 on dist-test-slave-hwlj"
server_key: "200c75251ab814236e6b23d7bc406306"
server_key_iv: "9574277f9ddf351079714ff13145796c"
server_key_version: "encryptionkey@0"
I20260926 20:54:46.498775 20652 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:54:46.499545 20667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:46.499811 20652 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:54:46.499943 20652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "2d5c640289a64c8a9aa1208376197f55"
format_stamp: "Formatted at 2026-09-26 20:54:46 on dist-test-slave-hwlj"
server_key: "200c75251ab814236e6b23d7bc406306"
server_key_iv: "9574277f9ddf351079714ff13145796c"
server_key_version: "encryptionkey@0"
I20260926 20:54:46.500033 20652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:54:46.503186 20652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:54:46.505131 20652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:54:46.505235 20652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:54:46.508997 20652 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40351
I20260926 20:54:46.509025 20719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40351 every 8 connection(s)
I20260926 20:54:46.509369 20652 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:54:46.510186 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 20652
I20260926 20:54:46.510277 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455725523895-392-0/minicluster-data/master-0/wal/instance
I20260926 20:54:46.510417 20720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:54:46.510505 392 external_mini_cluster.cc:1428] Setting key 0a265f0f30923e09444109fd966a492c
I20260926 20:54:46.512938 20723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:46.511056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56393 (local address 127.0.0.1:40351)
0926 20:54:46.511224 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:46.511231 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:46.511255 (+ 24us) server_negotiation.cc:407] Connection header received
0926 20:54:46.511700 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:46.511707 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:46.511733 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:46.511915 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:46.512574 (+ 659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:46.512577 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:46.512578 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:54:46.512581 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:46.512617 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:54:46.512627 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:46.512689 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:46.512775 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 20:54:46.512807 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":85,"threads_started":1}
I20260926 20:54:46.513952 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:54:46.514268 20720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d5c640289a64c8a9aa1208376197f55: Bootstrap starting.
I20260926 20:54:46.514863 20720 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d5c640289a64c8a9aa1208376197f55: Neither blocks nor log segments found. Creating new log.
I20260926 20:54:46.515203 20720 log.cc:824] T 00000000000000000000000000000000 P 2d5c640289a64c8a9aa1208376197f55: Log is configured to *not* fsync() on all Append() calls
I20260926 20:54:46.515905 20720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d5c640289a64c8a9aa1208376197f55: No bootstrap required, opened a new log
I20260926 20:54:46.516426 20723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:46.515318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57919 (local address 127.0.0.1:40351)
0926 20:54:46.515360 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:46.515362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:46.515466 (+ 104us) server_negotiation.cc:407] Connection header received
0926 20:54:46.515537 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:46.515537 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:46.515542 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:46.515584 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:46.516234 (+ 650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:46.516235 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:46.516236 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:54:46.516240 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:46.516252 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:54:46.516256 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:46.516299 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:46.516352 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 20:54:46.516366 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 20:54:46.517773 20720 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d5c640289a64c8a9aa1208376197f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d5c640289a64c8a9aa1208376197f55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40351 } }
I20260926 20:54:46.517925 20720 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d5c640289a64c8a9aa1208376197f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:54:46.517971 20720 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d5c640289a64c8a9aa1208376197f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d5c640289a64c8a9aa1208376197f55, State: Initialized, Role: FOLLOWER
I20260926 20:54:46.518088 20720 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d5c640289a64c8a9aa1208376197f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d5c640289a64c8a9aa1208376197f55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40351 } }
I20260926 20:54:46.518177 20720 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d5c640289a64c8a9aa1208376197f55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:54:46.518220 20720 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d5c640289a64c8a9aa1208376197f55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:54:46.518270 20720 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d5c640289a64c8a9aa1208376197f55 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:54:46.518764 20720 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d5c640289a64c8a9aa1208376197f55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d5c640289a64c8a9aa1208376197f55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40351 } }
I20260926 20:54:46.518877 20720 leader_election.cc:302] T 00000000000000000000000000000000 P 2d5c640289a64c8a9aa1208376197f55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d5c640289a64c8a9aa1208376197f55; no voters:
I20260926 20:54:46.519053 20720 leader_election.cc:288] T 00000000000000000000000000000000 P 2d5c640289a64c8a9aa1208376197f55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:54:46.519119 20730 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d5c640289a64c8a9aa1208376197f55 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:54:46.519299 20720 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d5c640289a64c8a9aa1208376197f55 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:54:46.519307 20730 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d5c640289a64c8a9aa1208376197f55 [term 1 LEADER]: Becoming Leader. State: Replica: 2d5c640289a64c8a9aa1208376197f55, State: Running, Role: LEADER
I20260926 20:54:46.519408 20730 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d5c640289a64c8a9aa1208376197f55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d5c640289a64c8a9aa1208376197f55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40351 } }
I20260926 20:54:46.519834 20731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d5c640289a64c8a9aa1208376197f55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d5c640289a64c8a9aa1208376197f55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d5c640289a64c8a9aa1208376197f55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40351 } } }
I20260926 20:54:46.519846 20732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d5c640289a64c8a9aa1208376197f55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d5c640289a64c8a9aa1208376197f55. Latest consensus state: current_term: 1 leader_uuid: "2d5c640289a64c8a9aa1208376197f55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d5c640289a64c8a9aa1208376197f55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40351 } } }
I20260926 20:54:46.519981 20731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d5c640289a64c8a9aa1208376197f55 [sys.catalog]: This master's current role is: LEADER
I20260926 20:54:46.520016 20732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d5c640289a64c8a9aa1208376197f55 [sys.catalog]: This master's current role is: LEADER
W20260926 20:54:46.521121 20746 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2d5c640289a64c8a9aa1208376197f55: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:54:46.521165 20746 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:54:46.521227 20739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:54:46.521368 20739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:54:46.522850 20739 catalog_manager.cc:1347] Generated new cluster ID: d0a59c5bc58248ec8525771c35cb32a0
I20260926 20:54:46.522907 20739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:54:46.528265 20739 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:54:46.528420 20739 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:54:46.534435 20739 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d5c640289a64c8a9aa1208376197f55: Generated new TSK 0
I20260926 20:54:46.534631 20739 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:54:46.542800 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 20652
2026-09-26T20:54:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1304 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455725523895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:54:46 dist-test-slave-hwlj krb5kdc[20751](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:54:46 dist-test-slave-hwlj krb5kdc[20751](info): set up 2 sockets
Sep 26 20:54:46 dist-test-slave-hwlj krb5kdc[20751](info): commencing operation
krb5kdc: starting...
W20260926 20:54:47.591948 392 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455725523895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455725523895-392-0/krb5kdc/test-user.keytab.
Sep 26 20:54:47 dist-test-slave-hwlj krb5kdc[20751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:54:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:54:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455725523895-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455725523895-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:54:47.711392 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44667
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455725523895-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:54:47.791426 20764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:54:47.791678 20764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:54:47.791718 20764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:54:47.792933 20764 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:54:47.793005 20764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:54:47.793033 20764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:54:47.793064 20764 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:54:47.793097 20764 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:54:47.794598 20764 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455725523895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455725523895-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:54:47.794870 20764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:54:47.795094 20764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:54:47.800392 20769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:47.800419 20770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:47.800392 20772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:47.800707 20764 server_base.cc:1034] running on GCE node
I20260926 20:54:47.800943 20764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:54:47.801286 20764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:54:47.802464 20764 hybrid_clock.cc:648] HybridClock initialized: now 1790456087802450 us; error 29 us; skew 500 ppm
Sep 26 20:54:47 dist-test-slave-hwlj krb5kdc[20751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:54:47.805171 20764 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:54:47.805670 20764 webserver.cc:460] Webserver started at http://127.0.0.1:44687/ using document root <none> and password file <none>
I20260926 20:54:47.805925 20764 fs_manager.cc:359] Metadata directory not provided
I20260926 20:54:47.805994 20764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:54:47.806164 20764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:54:47.807245 20764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455725523895-392-0/minicluster-data/master-0/data/instance:
uuid: "2fb23c838cf245aebcb93ad6dd8051fc"
format_stamp: "Formatted at 2026-09-26 20:54:47 on dist-test-slave-hwlj"
server_key: "483b033197c200ac5cee20037b9cb8ea"
server_key_iv: "771c1ccfd30a965a92a75ac85675042d"
server_key_version: "encryptionkey@0"
I20260926 20:54:47.807611 20764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455725523895-392-0/minicluster-data/master-0/wal/instance:
uuid: "2fb23c838cf245aebcb93ad6dd8051fc"
format_stamp: "Formatted at 2026-09-26 20:54:47 on dist-test-slave-hwlj"
server_key: "483b033197c200ac5cee20037b9cb8ea"
server_key_iv: "771c1ccfd30a965a92a75ac85675042d"
server_key_version: "encryptionkey@0"
I20260926 20:54:47.808962 20764 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:54:47.809736 20779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:47.809939 20764 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 20:54:47.810039 20764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "2fb23c838cf245aebcb93ad6dd8051fc"
format_stamp: "Formatted at 2026-09-26 20:54:47 on dist-test-slave-hwlj"
server_key: "483b033197c200ac5cee20037b9cb8ea"
server_key_iv: "771c1ccfd30a965a92a75ac85675042d"
server_key_version: "encryptionkey@0"
I20260926 20:54:47.810125 20764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:54:47.827539 20764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:54:47.829674 20764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:54:47.829823 20764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:54:47.833994 20764 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44667
I20260926 20:54:47.834061 20831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44667 every 8 connection(s)
I20260926 20:54:47.834403 20764 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:54:47.835255 20832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:54:47.836299 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 20764
I20260926 20:54:47.836412 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455725523895-392-0/minicluster-data/master-0/wal/instance
I20260926 20:54:47.836612 392 external_mini_cluster.cc:1428] Setting key 6211291bbde82a8676c40a2951b692c0
I20260926 20:54:47.838037 20832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fb23c838cf245aebcb93ad6dd8051fc: Bootstrap starting.
I20260926 20:54:47.838795 20832 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2fb23c838cf245aebcb93ad6dd8051fc: Neither blocks nor log segments found. Creating new log.
I20260926 20:54:47.839126 20832 log.cc:824] T 00000000000000000000000000000000 P 2fb23c838cf245aebcb93ad6dd8051fc: Log is configured to *not* fsync() on all Append() calls
I20260926 20:54:47.839891 20832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fb23c838cf245aebcb93ad6dd8051fc: No bootstrap required, opened a new log
I20260926 20:54:47.842213 20832 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2fb23c838cf245aebcb93ad6dd8051fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fb23c838cf245aebcb93ad6dd8051fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44667 } }
I20260926 20:54:47.842410 20832 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2fb23c838cf245aebcb93ad6dd8051fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:54:47.842453 20832 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2fb23c838cf245aebcb93ad6dd8051fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fb23c838cf245aebcb93ad6dd8051fc, State: Initialized, Role: FOLLOWER
I20260926 20:54:47.842587 20832 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fb23c838cf245aebcb93ad6dd8051fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fb23c838cf245aebcb93ad6dd8051fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44667 } }
I20260926 20:54:47.842675 20832 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2fb23c838cf245aebcb93ad6dd8051fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:54:47.842716 20832 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2fb23c838cf245aebcb93ad6dd8051fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:54:47.842770 20832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2fb23c838cf245aebcb93ad6dd8051fc [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:54:47.843544 20832 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2fb23c838cf245aebcb93ad6dd8051fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fb23c838cf245aebcb93ad6dd8051fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44667 } }
I20260926 20:54:47.843632 20832 leader_election.cc:302] T 00000000000000000000000000000000 P 2fb23c838cf245aebcb93ad6dd8051fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2fb23c838cf245aebcb93ad6dd8051fc; no voters:
I20260926 20:54:47.843835 20832 leader_election.cc:288] T 00000000000000000000000000000000 P 2fb23c838cf245aebcb93ad6dd8051fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:54:47.843860 20837 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2fb23c838cf245aebcb93ad6dd8051fc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:54:47.843982 20837 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2fb23c838cf245aebcb93ad6dd8051fc [term 1 LEADER]: Becoming Leader. State: Replica: 2fb23c838cf245aebcb93ad6dd8051fc, State: Running, Role: LEADER
I20260926 20:54:47.844091 20832 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fb23c838cf245aebcb93ad6dd8051fc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:54:47.844061 20837 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fb23c838cf245aebcb93ad6dd8051fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fb23c838cf245aebcb93ad6dd8051fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44667 } }
Sep 26 20:54:47 dist-test-slave-hwlj krb5kdc[20751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 20:54:47.844476 20837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fb23c838cf245aebcb93ad6dd8051fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fb23c838cf245aebcb93ad6dd8051fc. Latest consensus state: current_term: 1 leader_uuid: "2fb23c838cf245aebcb93ad6dd8051fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fb23c838cf245aebcb93ad6dd8051fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44667 } } }
I20260926 20:54:47.844583 20837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fb23c838cf245aebcb93ad6dd8051fc [sys.catalog]: This master's current role is: LEADER
I20260926 20:54:47.844501 20839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fb23c838cf245aebcb93ad6dd8051fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2fb23c838cf245aebcb93ad6dd8051fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fb23c838cf245aebcb93ad6dd8051fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44667 } } }
I20260926 20:54:47.844738 20839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fb23c838cf245aebcb93ad6dd8051fc [sys.catalog]: This master's current role is: LEADER
I20260926 20:54:47.845458 20844 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:54:47.846264 20844 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:54:47.847795 20844 catalog_manager.cc:1347] Generated new cluster ID: 21e8c5e86c3c490a91b7e66f8eb97b41
I20260926 20:54:47.847844 20844 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:54:47.849076 20835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:47.837213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.173:46995 (local address 127.0.0.1:44667)
0926 20:54:47.837372 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:47.837380 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:47.837408 (+ 28us) server_negotiation.cc:407] Connection header received
0926 20:54:47.837912 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:47.837919 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:47.837968 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:47.838083 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:47.838961 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:47.840187 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:47.841281 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:47.841719 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:47.841806 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:47.844825 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:47.844830 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:47.844831 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:54:47.844845 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:47.846966 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:47.847538 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:47.847556 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:47.847556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:47.847674 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:47.848034 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:47.848035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:47.848035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:47.848315 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:47.848457 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:47.848696 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 20:54:47.848791 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":80,"threads_started":1}
I20260926 20:54:47.850029 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:54:47.857635 20844 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:54:47.858083 20844 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:54:47.859517 20835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:47.851793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.173:39533 (local address 127.0.0.1:44667)
0926 20:54:47.851831 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:47.851832 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:47.851930 (+ 98us) server_negotiation.cc:407] Connection header received
0926 20:54:47.851975 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:47.851976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:47.851987 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:47.852045 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:47.852636 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:47.853816 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:47.854788 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:47.855113 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:47.855200 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:47.856608 (+ 1408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:47.856609 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:47.856610 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:54:47.856616 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:47.858312 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:47.858727 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:47.858728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:47.858729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:47.858789 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:47.859015 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:47.859016 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:47.859016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:47.859148 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:47.859251 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:47.859364 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 20:54:47.859426 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 20:54:47.865026 20844 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2fb23c838cf245aebcb93ad6dd8051fc: Generated new TSK 0
I20260926 20:54:47.865206 20844 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:54:47.869763 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 20764
W20260926 20:54:47.874037 20749 connection.cc:535] client connection to 127.0.0.1:44667 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44667): Connection reset by peer (error 104)
2026-09-26T20:54:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1326 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455725523895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:54:47 dist-test-slave-hwlj krb5kdc[20863](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:54:47 dist-test-slave-hwlj krb5kdc[20863](info): set up 2 sockets
Sep 26 20:54:47 dist-test-slave-hwlj krb5kdc[20863](info): commencing operation
krb5kdc: starting...
W20260926 20:54:48.903997 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455725523895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455725523895-392-0/krb5kdc/test-user.keytab.
Sep 26 20:54:48 dist-test-slave-hwlj krb5kdc[20863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:54:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:54:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455725523895-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455725523895-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:54:49.029472 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34325
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455725523895-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:54:49.108920 20876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:54:49.109126 20876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:54:49.109150 20876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:54:49.110327 20876 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:54:49.110375 20876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:54:49.110390 20876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:54:49.110402 20876 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:54:49.110457 20876 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:54:49.111969 20876 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455725523895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455725523895-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:54:49.112236 20876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:54:49.112471 20876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:54:49.117560 20881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:49.117568 20882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:49.117571 20884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:49.117799 20876 server_base.cc:1034] running on GCE node
I20260926 20:54:49.118112 20876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:54:49.118431 20876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:54:49.119650 20876 hybrid_clock.cc:648] HybridClock initialized: now 1790456089119639 us; error 38 us; skew 500 ppm
Sep 26 20:54:49 dist-test-slave-hwlj krb5kdc[20863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456089, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:54:49.122390 20876 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:54:49.123060 20876 webserver.cc:460] Webserver started at http://127.0.0.1:33751/ using document root <none> and password file <none>
I20260926 20:54:49.123373 20876 fs_manager.cc:359] Metadata directory not provided
I20260926 20:54:49.123447 20876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:54:49.123600 20876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:54:49.124802 20876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455725523895-392-0/minicluster-data/master-0/data/instance:
uuid: "34fad7a486974d228066a0537593f343"
format_stamp: "Formatted at 2026-09-26 20:54:49 on dist-test-slave-hwlj"
server_key: "4b5e9726b4c7c010b84df566a3ba60fd"
server_key_iv: "beead199e26432ff4fd65a1182fc2ac3"
server_key_version: "encryptionkey@0"
I20260926 20:54:49.125110 20876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455725523895-392-0/minicluster-data/master-0/wal/instance:
uuid: "34fad7a486974d228066a0537593f343"
format_stamp: "Formatted at 2026-09-26 20:54:49 on dist-test-slave-hwlj"
server_key: "4b5e9726b4c7c010b84df566a3ba60fd"
server_key_iv: "beead199e26432ff4fd65a1182fc2ac3"
server_key_version: "encryptionkey@0"
I20260926 20:54:49.126406 20876 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:54:49.127141 20891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:49.127337 20876 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:54:49.127439 20876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "34fad7a486974d228066a0537593f343"
format_stamp: "Formatted at 2026-09-26 20:54:49 on dist-test-slave-hwlj"
server_key: "4b5e9726b4c7c010b84df566a3ba60fd"
server_key_iv: "beead199e26432ff4fd65a1182fc2ac3"
server_key_version: "encryptionkey@0"
I20260926 20:54:49.127533 20876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:54:49.141844 20876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:54:49.144220 20876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:54:49.144392 20876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:54:49.148228 20876 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34325
I20260926 20:54:49.148528 20943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34325 every 8 connection(s)
I20260926 20:54:49.148629 20876 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:54:49.149327 20944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:54:49.151566 20944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34fad7a486974d228066a0537593f343: Bootstrap starting.
I20260926 20:54:49.152171 20944 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34fad7a486974d228066a0537593f343: Neither blocks nor log segments found. Creating new log.
I20260926 20:54:49.152405 20944 log.cc:824] T 00000000000000000000000000000000 P 34fad7a486974d228066a0537593f343: Log is configured to *not* fsync() on all Append() calls
I20260926 20:54:49.153066 20944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34fad7a486974d228066a0537593f343: No bootstrap required, opened a new log
I20260926 20:54:49.154278 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 20876
I20260926 20:54:49.154405 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455725523895-392-0/minicluster-data/master-0/wal/instance
I20260926 20:54:49.154596 392 external_mini_cluster.cc:1428] Setting key 6174bd0c9eedea3a9267df4c89904ad7
I20260926 20:54:49.154891 20944 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34fad7a486974d228066a0537593f343 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34fad7a486974d228066a0537593f343" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34325 } }
I20260926 20:54:49.155083 20944 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34fad7a486974d228066a0537593f343 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:54:49.155133 20944 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34fad7a486974d228066a0537593f343 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34fad7a486974d228066a0537593f343, State: Initialized, Role: FOLLOWER
I20260926 20:54:49.155270 20944 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34fad7a486974d228066a0537593f343 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34fad7a486974d228066a0537593f343" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34325 } }
I20260926 20:54:49.155361 20944 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34fad7a486974d228066a0537593f343 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:54:49.155395 20944 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34fad7a486974d228066a0537593f343 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:54:49.155441 20944 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34fad7a486974d228066a0537593f343 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:54:49.156107 20944 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34fad7a486974d228066a0537593f343 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34fad7a486974d228066a0537593f343" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34325 } }
I20260926 20:54:49.156224 20944 leader_election.cc:302] T 00000000000000000000000000000000 P 34fad7a486974d228066a0537593f343 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34fad7a486974d228066a0537593f343; no voters:
I20260926 20:54:49.156399 20944 leader_election.cc:288] T 00000000000000000000000000000000 P 34fad7a486974d228066a0537593f343 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:54:49.156450 20949 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34fad7a486974d228066a0537593f343 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:54:49.156589 20949 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34fad7a486974d228066a0537593f343 [term 1 LEADER]: Becoming Leader. State: Replica: 34fad7a486974d228066a0537593f343, State: Running, Role: LEADER
I20260926 20:54:49.156651 20944 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34fad7a486974d228066a0537593f343 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:54:49.156680 20949 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34fad7a486974d228066a0537593f343 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34fad7a486974d228066a0537593f343" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34325 } }
I20260926 20:54:49.157061 20949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34fad7a486974d228066a0537593f343 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34fad7a486974d228066a0537593f343. Latest consensus state: current_term: 1 leader_uuid: "34fad7a486974d228066a0537593f343" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34fad7a486974d228066a0537593f343" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34325 } } }
I20260926 20:54:49.157151 20949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34fad7a486974d228066a0537593f343 [sys.catalog]: This master's current role is: LEADER
I20260926 20:54:49.157071 20950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34fad7a486974d228066a0537593f343 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34fad7a486974d228066a0537593f343" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34fad7a486974d228066a0537593f343" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34325 } } }
I20260926 20:54:49.157290 20950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34fad7a486974d228066a0537593f343 [sys.catalog]: This master's current role is: LEADER
I20260926 20:54:49.157377 20958 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:54:49.157878 20958 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:54:49.159477 20958 catalog_manager.cc:1347] Generated new cluster ID: 9b328ebc25d94e62bdea6825cb6a4c1e
I20260926 20:54:49.159528 20958 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:54:49.161016 20948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:49.155147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.173:50997 (local address 127.0.0.1:34325)
0926 20:54:49.155368 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:49.155378 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:49.155413 (+ 35us) server_negotiation.cc:407] Connection header received
0926 20:54:49.155849 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:49.155856 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:49.155906 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:49.156017 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:49.156304 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:49.157790 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:49.158852 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:49.159448 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:49.159904 (+ 456us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:49.160325 (+ 421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:49.160329 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:49.160329 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:54:49.160541 (+ 212us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:49.160577 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:54:49.160590 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:49.160670 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:49.160802 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 20:54:49.160827 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":85,"threads_started":1}
I20260926 20:54:49.162014 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:54:49.166867 20948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:49.163397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.173:60177 (local address 127.0.0.1:34325)
0926 20:54:49.163433 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:49.163434 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:49.163562 (+ 128us) server_negotiation.cc:407] Connection header received
0926 20:54:49.163604 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:49.163605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:49.163616 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:49.163673 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:49.163827 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:49.164565 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:49.165552 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:49.165862 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:49.165934 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:49.166620 (+ 686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:49.166622 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:49.166622 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:54:49.166626 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:49.166640 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:54:49.166644 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:49.166709 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:49.166783 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 20:54:49.166796 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 20:54:49.172806 20958 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:54:49.173491 20958 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:54:49.181340 20958 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34fad7a486974d228066a0537593f343: Generated new TSK 0
I20260926 20:54:49.181538 20958 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:54:49.183709 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 20876
W20260926 20:54:49.187690 20861 connection.cc:535] client connection to 127.0.0.1:34325 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34325): Connection reset by peer (error 104)
2026-09-26T20:54:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1313 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455725523895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:54:49 dist-test-slave-hwlj krb5kdc[20975](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:54:49 dist-test-slave-hwlj krb5kdc[20975](info): set up 2 sockets
Sep 26 20:54:49 dist-test-slave-hwlj krb5kdc[20975](info): commencing operation
krb5kdc: starting...
W20260926 20:54:50.211860 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455725523895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455725523895-392-0/krb5kdc/test-user.keytab.
Sep 26 20:54:50 dist-test-slave-hwlj krb5kdc[20975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:54:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:54:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455725523895-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455725523895-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:54:50.332813 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43987
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455725523895-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:54:50.416076 20988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:54:50.416323 20988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:54:50.416366 20988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:54:50.417554 20988 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:54:50.417624 20988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:54:50.417649 20988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:54:50.417670 20988 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:54:50.417703 20988 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:54:50.419203 20988 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455725523895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455725523895-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:54:50.419486 20988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:54:50.419718 20988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:54:50.424783 20994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:50.424778 20993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:50.424805 20996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:50.424974 20988 server_base.cc:1034] running on GCE node
I20260926 20:54:50.425302 20988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:54:50.425606 20988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:54:50.426800 20988 hybrid_clock.cc:648] HybridClock initialized: now 1790456090426771 us; error 38 us; skew 500 ppm
Sep 26 20:54:50 dist-test-slave-hwlj krb5kdc[20975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456090, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:54:50.429447 20988 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:54:50.429926 20988 webserver.cc:460] Webserver started at http://127.0.0.1:35731/ using document root <none> and password file <none>
I20260926 20:54:50.430174 20988 fs_manager.cc:359] Metadata directory not provided
I20260926 20:54:50.430243 20988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:54:50.430399 20988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:54:50.431488 20988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455725523895-392-0/minicluster-data/master-0/data/instance:
uuid: "9e8c34208f6341a395c6a5c9e91a4dc5"
format_stamp: "Formatted at 2026-09-26 20:54:50 on dist-test-slave-hwlj"
server_key: "1ab47c2cc092d34b7248317589aed392"
server_key_iv: "dfbe957f4a5b50c94393e0f06071ad08"
server_key_version: "encryptionkey@0"
I20260926 20:54:50.431905 20988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455725523895-392-0/minicluster-data/master-0/wal/instance:
uuid: "9e8c34208f6341a395c6a5c9e91a4dc5"
format_stamp: "Formatted at 2026-09-26 20:54:50 on dist-test-slave-hwlj"
server_key: "1ab47c2cc092d34b7248317589aed392"
server_key_iv: "dfbe957f4a5b50c94393e0f06071ad08"
server_key_version: "encryptionkey@0"
I20260926 20:54:50.433274 20988 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:54:50.434013 21003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:50.434306 20988 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:54:50.434408 20988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "9e8c34208f6341a395c6a5c9e91a4dc5"
format_stamp: "Formatted at 2026-09-26 20:54:50 on dist-test-slave-hwlj"
server_key: "1ab47c2cc092d34b7248317589aed392"
server_key_iv: "dfbe957f4a5b50c94393e0f06071ad08"
server_key_version: "encryptionkey@0"
I20260926 20:54:50.434490 20988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:54:50.437878 20988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:54:50.439953 20988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:54:50.440104 20988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:54:50.444136 20988 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43987
I20260926 20:54:50.444170 21055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43987 every 8 connection(s)
I20260926 20:54:50.444531 20988 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:54:50.445525 21056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:54:50.447603 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 20988
I20260926 20:54:50.447701 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455725523895-392-0/minicluster-data/master-0/wal/instance
I20260926 20:54:50.447957 392 external_mini_cluster.cc:1428] Setting key 309e5606eab8f96158621b5fa384f9b8
I20260926 20:54:50.447976 21056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e8c34208f6341a395c6a5c9e91a4dc5: Bootstrap starting.
I20260926 20:54:50.448758 21056 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e8c34208f6341a395c6a5c9e91a4dc5: Neither blocks nor log segments found. Creating new log.
I20260926 20:54:50.449093 21056 log.cc:824] T 00000000000000000000000000000000 P 9e8c34208f6341a395c6a5c9e91a4dc5: Log is configured to *not* fsync() on all Append() calls
I20260926 20:54:50.449869 21056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e8c34208f6341a395c6a5c9e91a4dc5: No bootstrap required, opened a new log
I20260926 20:54:50.450876 21059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:50.448553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.173:40029 (local address 127.0.0.1:43987)
0926 20:54:50.448794 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:50.448804 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:50.448861 (+ 57us) server_negotiation.cc:407] Connection header received
0926 20:54:50.449522 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:50.449532 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:50.449565 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:50.449714 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:50.450401 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:50.450405 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:50.450406 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:54:50.450410 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:50.450453 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:54:50.450467 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:50.450568 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:50.450640 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 20:54:50.450685 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":106,"threads_started":1}
I20260926 20:54:50.452030 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:54:50.452402 21056 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e8c34208f6341a395c6a5c9e91a4dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e8c34208f6341a395c6a5c9e91a4dc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43987 } }
I20260926 20:54:50.452585 21056 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e8c34208f6341a395c6a5c9e91a4dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:54:50.452622 21056 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e8c34208f6341a395c6a5c9e91a4dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e8c34208f6341a395c6a5c9e91a4dc5, State: Initialized, Role: FOLLOWER
I20260926 20:54:50.452764 21056 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e8c34208f6341a395c6a5c9e91a4dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e8c34208f6341a395c6a5c9e91a4dc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43987 } }
I20260926 20:54:50.452875 21056 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e8c34208f6341a395c6a5c9e91a4dc5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:54:50.452908 21056 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e8c34208f6341a395c6a5c9e91a4dc5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:54:50.452956 21056 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e8c34208f6341a395c6a5c9e91a4dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:54:50.453737 21056 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e8c34208f6341a395c6a5c9e91a4dc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e8c34208f6341a395c6a5c9e91a4dc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43987 } }
I20260926 20:54:50.453871 21056 leader_election.cc:302] T 00000000000000000000000000000000 P 9e8c34208f6341a395c6a5c9e91a4dc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e8c34208f6341a395c6a5c9e91a4dc5; no voters:
I20260926 20:54:50.454058 21056 leader_election.cc:288] T 00000000000000000000000000000000 P 9e8c34208f6341a395c6a5c9e91a4dc5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:54:50.454079 21066 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e8c34208f6341a395c6a5c9e91a4dc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:54:50.454187 21066 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e8c34208f6341a395c6a5c9e91a4dc5 [term 1 LEADER]: Becoming Leader. State: Replica: 9e8c34208f6341a395c6a5c9e91a4dc5, State: Running, Role: LEADER
I20260926 20:54:50.454308 21056 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e8c34208f6341a395c6a5c9e91a4dc5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:54:50.454300 21066 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e8c34208f6341a395c6a5c9e91a4dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e8c34208f6341a395c6a5c9e91a4dc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43987 } }
I20260926 20:54:50.454761 21059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:50.453632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.173:54043 (local address 127.0.0.1:43987)
0926 20:54:50.453672 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:50.453674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:50.453826 (+ 152us) server_negotiation.cc:407] Connection header received
0926 20:54:50.453903 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:50.453904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:50.453909 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:50.453961 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:50.454548 (+ 587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:50.454549 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:50.454550 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:54:50.454554 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:50.454568 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:54:50.454572 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:50.454621 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:50.454667 (+ 46us) server_negotiation.cc:299] Negotiation successful
0926 20:54:50.454682 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 20:54:50.455503 21067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e8c34208f6341a395c6a5c9e91a4dc5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e8c34208f6341a395c6a5c9e91a4dc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e8c34208f6341a395c6a5c9e91a4dc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43987 } } }
I20260926 20:54:50.455519 21068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e8c34208f6341a395c6a5c9e91a4dc5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e8c34208f6341a395c6a5c9e91a4dc5. Latest consensus state: current_term: 1 leader_uuid: "9e8c34208f6341a395c6a5c9e91a4dc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e8c34208f6341a395c6a5c9e91a4dc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43987 } } }
I20260926 20:54:50.455775 21067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e8c34208f6341a395c6a5c9e91a4dc5 [sys.catalog]: This master's current role is: LEADER
I20260926 20:54:50.455780 21068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e8c34208f6341a395c6a5c9e91a4dc5 [sys.catalog]: This master's current role is: LEADER
W20260926 20:54:50.456256 21080 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9e8c34208f6341a395c6a5c9e91a4dc5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:54:50.456368 21080 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:54:50.456463 21081 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:54:50.456632 21081 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:54:50.458206 21081 catalog_manager.cc:1347] Generated new cluster ID: a2f20fbf993b4a9185342c2927618c09
I20260926 20:54:50.458271 21081 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:54:50.465914 21081 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:54:50.466118 21081 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:54:50.468880 21081 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e8c34208f6341a395c6a5c9e91a4dc5: Generated new TSK 0
I20260926 20:54:50.469048 21081 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:54:50.479097 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 20988
2026-09-26T20:54:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1295 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455725523895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:54:50 dist-test-slave-hwlj krb5kdc[21087](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:54:50 dist-test-slave-hwlj krb5kdc[21087](info): set up 2 sockets
Sep 26 20:54:50 dist-test-slave-hwlj krb5kdc[21087](info): commencing operation
krb5kdc: starting...
W20260926 20:54:51.515949 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455725523895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455725523895-392-0/krb5kdc/test-user.keytab.
Sep 26 20:54:51 dist-test-slave-hwlj krb5kdc[21087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:54:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:54:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455725523895-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455725523895-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:54:51.634820 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43407
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455725523895-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43407
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:54:51.714646 21100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:54:51.714856 21100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:54:51.714879 21100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:54:51.716001 21100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:54:51.716037 21100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:54:51.716050 21100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:54:51.716063 21100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:54:51.716075 21100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:54:51.717442 21100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455725523895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43407
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455725523895-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:54:51.717635 21100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:54:51.717797 21100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:54:51.722625 21106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:51.722610 21108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:51.722950 21100 server_base.cc:1034] running on GCE node
W20260926 20:54:51.722644 21105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:51.723244 21100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:54:51.723505 21100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:54:51.724658 21100 hybrid_clock.cc:648] HybridClock initialized: now 1790456091724631 us; error 26 us; skew 500 ppm
Sep 26 20:54:51 dist-test-slave-hwlj krb5kdc[21087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456091, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:54:51.731149 21100 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:54:51.731655 21100 webserver.cc:460] Webserver started at http://127.0.0.1:36437/ using document root <none> and password file <none>
I20260926 20:54:51.731909 21100 fs_manager.cc:359] Metadata directory not provided
I20260926 20:54:51.731979 21100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:54:51.732113 21100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:54:51.733179 21100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455725523895-392-0/minicluster-data/master-0/data/instance:
uuid: "e8bac7c6dc4b4de2b726e2035783f38f"
format_stamp: "Formatted at 2026-09-26 20:54:51 on dist-test-slave-hwlj"
server_key: "a460e0feceeee21a5c460e73489df33f"
server_key_iv: "b6f367e8d1059caf1728c9165af4bb4f"
server_key_version: "encryptionkey@0"
I20260926 20:54:51.733537 21100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455725523895-392-0/minicluster-data/master-0/wal/instance:
uuid: "e8bac7c6dc4b4de2b726e2035783f38f"
format_stamp: "Formatted at 2026-09-26 20:54:51 on dist-test-slave-hwlj"
server_key: "a460e0feceeee21a5c460e73489df33f"
server_key_iv: "b6f367e8d1059caf1728c9165af4bb4f"
server_key_version: "encryptionkey@0"
I20260926 20:54:51.734865 21100 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:54:51.735605 21115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:51.735821 21100 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:54:51.735924 21100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "e8bac7c6dc4b4de2b726e2035783f38f"
format_stamp: "Formatted at 2026-09-26 20:54:51 on dist-test-slave-hwlj"
server_key: "a460e0feceeee21a5c460e73489df33f"
server_key_iv: "b6f367e8d1059caf1728c9165af4bb4f"
server_key_version: "encryptionkey@0"
I20260926 20:54:51.736011 21100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:54:51.751140 21100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:54:51.753517 21100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:54:51.753682 21100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:54:51.757759 21100 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43407
I20260926 20:54:51.757792 21167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43407 every 8 connection(s)
I20260926 20:54:51.758176 21100 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:54:51.759014 21168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:54:51.759577 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 21100
I20260926 20:54:51.759702 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455725523895-392-0/minicluster-data/master-0/wal/instance
I20260926 20:54:51.759932 392 external_mini_cluster.cc:1428] Setting key 8e4acad4e4c4c830766c245962b7d915
I20260926 20:54:51.762310 21168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8bac7c6dc4b4de2b726e2035783f38f: Bootstrap starting.
I20260926 20:54:51.762903 21168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8bac7c6dc4b4de2b726e2035783f38f: Neither blocks nor log segments found. Creating new log.
I20260926 20:54:51.763253 21168 log.cc:824] T 00000000000000000000000000000000 P e8bac7c6dc4b4de2b726e2035783f38f: Log is configured to *not* fsync() on all Append() calls
I20260926 20:54:51.764029 21168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8bac7c6dc4b4de2b726e2035783f38f: No bootstrap required, opened a new log
I20260926 20:54:51.766601 21168 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8bac7c6dc4b4de2b726e2035783f38f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8bac7c6dc4b4de2b726e2035783f38f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43407 } }
Sep 26 20:54:51 dist-test-slave-hwlj krb5kdc[21087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 20:54:51.766737 21168 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8bac7c6dc4b4de2b726e2035783f38f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:54:51.766803 21168 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8bac7c6dc4b4de2b726e2035783f38f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8bac7c6dc4b4de2b726e2035783f38f, State: Initialized, Role: FOLLOWER
I20260926 20:54:51.766937 21168 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8bac7c6dc4b4de2b726e2035783f38f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8bac7c6dc4b4de2b726e2035783f38f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43407 } }
I20260926 20:54:51.766997 21168 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8bac7c6dc4b4de2b726e2035783f38f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:54:51.767019 21168 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8bac7c6dc4b4de2b726e2035783f38f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:54:51.767091 21168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8bac7c6dc4b4de2b726e2035783f38f [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:54:51.767678 21168 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8bac7c6dc4b4de2b726e2035783f38f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8bac7c6dc4b4de2b726e2035783f38f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43407 } }
I20260926 20:54:51.767812 21168 leader_election.cc:302] T 00000000000000000000000000000000 P e8bac7c6dc4b4de2b726e2035783f38f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8bac7c6dc4b4de2b726e2035783f38f; no voters:
I20260926 20:54:51.768011 21168 leader_election.cc:288] T 00000000000000000000000000000000 P e8bac7c6dc4b4de2b726e2035783f38f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:54:51.768040 21173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8bac7c6dc4b4de2b726e2035783f38f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:54:51.768157 21173 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8bac7c6dc4b4de2b726e2035783f38f [term 1 LEADER]: Becoming Leader. State: Replica: e8bac7c6dc4b4de2b726e2035783f38f, State: Running, Role: LEADER
I20260926 20:54:51.768291 21168 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8bac7c6dc4b4de2b726e2035783f38f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:54:51.768272 21173 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8bac7c6dc4b4de2b726e2035783f38f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8bac7c6dc4b4de2b726e2035783f38f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43407 } }
I20260926 20:54:51.768709 21173 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8bac7c6dc4b4de2b726e2035783f38f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8bac7c6dc4b4de2b726e2035783f38f. Latest consensus state: current_term: 1 leader_uuid: "e8bac7c6dc4b4de2b726e2035783f38f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8bac7c6dc4b4de2b726e2035783f38f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43407 } } }
I20260926 20:54:51.768741 21174 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8bac7c6dc4b4de2b726e2035783f38f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8bac7c6dc4b4de2b726e2035783f38f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8bac7c6dc4b4de2b726e2035783f38f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43407 } } }
I20260926 20:54:51.768795 21173 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8bac7c6dc4b4de2b726e2035783f38f [sys.catalog]: This master's current role is: LEADER
I20260926 20:54:51.768821 21174 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8bac7c6dc4b4de2b726e2035783f38f [sys.catalog]: This master's current role is: LEADER
I20260926 20:54:51.768960 21180 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:54:51.769428 21180 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:54:51.771166 21180 catalog_manager.cc:1347] Generated new cluster ID: 19aa529610314cf9a066bfe1cdc08f31
I20260926 20:54:51.771220 21180 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:54:51.771484 21171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:51.760500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.173:40673 (local address 127.0.0.1:43407)
0926 20:54:51.760658 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:51.760668 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:51.760706 (+ 38us) server_negotiation.cc:407] Connection header received
0926 20:54:51.761110 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:51.761117 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:51.761181 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:51.761300 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:51.762096 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:51.763028 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:51.764074 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:51.764523 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:51.764605 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:51.767315 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:51.767319 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:51.767319 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:54:51.767329 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:51.769473 (+ 2144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:51.769938 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:51.769942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:51.769942 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:51.770045 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:51.770317 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:51.770318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:51.770318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:51.770622 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:51.770772 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:51.771182 (+ 410us) server_negotiation.cc:299] Negotiation successful
0926 20:54:51.771292 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":80,"threads_started":1}
I20260926 20:54:51.772455 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:54:51.781176 21171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:51.773958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.173:47967 (local address 127.0.0.1:43407)
0926 20:54:51.773992 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:51.773994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:51.774052 (+ 58us) server_negotiation.cc:407] Connection header received
0926 20:54:51.774108 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:51.774109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:51.774118 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:51.774191 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:51.774761 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:51.775567 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:51.776572 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:51.776918 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:51.776990 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:51.778306 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:51.778307 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:51.778308 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:54:51.778315 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:51.779927 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:51.780326 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:51.780327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:51.780328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:51.780391 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:51.780648 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:51.780650 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:51.780650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:51.780781 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:51.780891 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:51.781019 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 20:54:51.781084 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 20:54:51.801208 21180 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:54:51.801982 21180 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:54:51.808130 21180 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8bac7c6dc4b4de2b726e2035783f38f: Generated new TSK 0
I20260926 20:54:51.808329 21180 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:54:51.815948 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 21100
W20260926 20:54:51.819855 21085 connection.cc:535] client connection to 127.0.0.1:43407 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43407): Connection reset by peer (error 104)
2026-09-26T20:54:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1336 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455725523895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:54:51 dist-test-slave-hwlj krb5kdc[21199](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:54:51 dist-test-slave-hwlj krb5kdc[21199](info): set up 2 sockets
Sep 26 20:54:51 dist-test-slave-hwlj krb5kdc[21199](info): commencing operation
krb5kdc: starting...
W20260926 20:54:52.847911 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455725523895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455725523895-392-0/krb5kdc/test-user.keytab.
Sep 26 20:54:52 dist-test-slave-hwlj krb5kdc[21199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:54:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:54:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455725523895-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455725523895-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:54:52.972779 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43097
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455725523895-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43097
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:54:53.053740 21212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:54:53.053951 21212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:54:53.053975 21212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:54:53.055092 21212 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:54:53.055133 21212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:54:53.055147 21212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:54:53.055160 21212 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:54:53.055172 21212 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:54:53.056624 21212 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455725523895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43097
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455725523895-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:54:53.056814 21212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:54:53.056979 21212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:54:53.061822 21220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:53.061954 21218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:53.061965 21217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:53.062150 21212 server_base.cc:1034] running on GCE node
I20260926 20:54:53.062367 21212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:54:53.062602 21212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:54:53.063829 21212 hybrid_clock.cc:648] HybridClock initialized: now 1790456093063804 us; error 37 us; skew 500 ppm
Sep 26 20:54:53 dist-test-slave-hwlj krb5kdc[21199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:54:53.066655 21212 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:54:53.067111 21212 webserver.cc:460] Webserver started at http://127.0.0.1:39219/ using document root <none> and password file <none>
I20260926 20:54:53.067353 21212 fs_manager.cc:359] Metadata directory not provided
I20260926 20:54:53.067444 21212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:54:53.067592 21212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:54:53.068768 21212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455725523895-392-0/minicluster-data/master-0/data/instance:
uuid: "889fb6342df04f799e959f47b74026bc"
format_stamp: "Formatted at 2026-09-26 20:54:53 on dist-test-slave-hwlj"
server_key: "10d7c47f8a19227a8950d3366e849a5b"
server_key_iv: "a181920b567f99f114576525fa1a02f4"
server_key_version: "encryptionkey@0"
I20260926 20:54:53.069135 21212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455725523895-392-0/minicluster-data/master-0/wal/instance:
uuid: "889fb6342df04f799e959f47b74026bc"
format_stamp: "Formatted at 2026-09-26 20:54:53 on dist-test-slave-hwlj"
server_key: "10d7c47f8a19227a8950d3366e849a5b"
server_key_iv: "a181920b567f99f114576525fa1a02f4"
server_key_version: "encryptionkey@0"
I20260926 20:54:53.070499 21212 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:54:53.071305 21227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:53.071659 21212 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:54:53.071781 21212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "889fb6342df04f799e959f47b74026bc"
format_stamp: "Formatted at 2026-09-26 20:54:53 on dist-test-slave-hwlj"
server_key: "10d7c47f8a19227a8950d3366e849a5b"
server_key_iv: "a181920b567f99f114576525fa1a02f4"
server_key_version: "encryptionkey@0"
I20260926 20:54:53.071870 21212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:54:53.096076 21212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:54:53.098505 21212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:54:53.098681 21212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:54:53.102650 21212 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43097
I20260926 20:54:53.102722 21279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43097 every 8 connection(s)
I20260926 20:54:53.103085 21212 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:54:53.104105 21280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:54:53.106360 21280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 889fb6342df04f799e959f47b74026bc: Bootstrap starting.
I20260926 20:54:53.106952 21280 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 889fb6342df04f799e959f47b74026bc: Neither blocks nor log segments found. Creating new log.
I20260926 20:54:53.107232 21280 log.cc:824] T 00000000000000000000000000000000 P 889fb6342df04f799e959f47b74026bc: Log is configured to *not* fsync() on all Append() calls
I20260926 20:54:53.107955 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 21212
I20260926 20:54:53.108064 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455725523895-392-0/minicluster-data/master-0/wal/instance
I20260926 20:54:53.108091 21280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 889fb6342df04f799e959f47b74026bc: No bootstrap required, opened a new log
I20260926 20:54:53.108278 392 external_mini_cluster.cc:1428] Setting key 3afdee55a0330850a37af91c44aeb071
I20260926 20:54:53.109887 21280 raft_consensus.cc:348] T 00000000000000000000000000000000 P 889fb6342df04f799e959f47b74026bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "889fb6342df04f799e959f47b74026bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43097 } }
I20260926 20:54:53.110023 21280 raft_consensus.cc:374] T 00000000000000000000000000000000 P 889fb6342df04f799e959f47b74026bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:54:53.110044 21280 raft_consensus.cc:729] T 00000000000000000000000000000000 P 889fb6342df04f799e959f47b74026bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 889fb6342df04f799e959f47b74026bc, State: Initialized, Role: FOLLOWER
I20260926 20:54:53.110188 21280 consensus_queue.cc:260] T 00000000000000000000000000000000 P 889fb6342df04f799e959f47b74026bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "889fb6342df04f799e959f47b74026bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43097 } }
I20260926 20:54:53.110281 21280 raft_consensus.cc:388] T 00000000000000000000000000000000 P 889fb6342df04f799e959f47b74026bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:54:53.110313 21280 raft_consensus.cc:482] T 00000000000000000000000000000000 P 889fb6342df04f799e959f47b74026bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:54:53.110360 21280 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 889fb6342df04f799e959f47b74026bc [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:54:53.110906 21280 raft_consensus.cc:504] T 00000000000000000000000000000000 P 889fb6342df04f799e959f47b74026bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "889fb6342df04f799e959f47b74026bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43097 } }
I20260926 20:54:53.110991 21280 leader_election.cc:302] T 00000000000000000000000000000000 P 889fb6342df04f799e959f47b74026bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 889fb6342df04f799e959f47b74026bc; no voters:
I20260926 20:54:53.111207 21280 leader_election.cc:288] T 00000000000000000000000000000000 P 889fb6342df04f799e959f47b74026bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:54:53.111225 21285 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 889fb6342df04f799e959f47b74026bc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:54:53.111335 21285 raft_consensus.cc:686] T 00000000000000000000000000000000 P 889fb6342df04f799e959f47b74026bc [term 1 LEADER]: Becoming Leader. State: Replica: 889fb6342df04f799e959f47b74026bc, State: Running, Role: LEADER
I20260926 20:54:53.111469 21280 sys_catalog.cc:564] T 00000000000000000000000000000000 P 889fb6342df04f799e959f47b74026bc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:54:53.111469 21285 consensus_queue.cc:237] T 00000000000000000000000000000000 P 889fb6342df04f799e959f47b74026bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "889fb6342df04f799e959f47b74026bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43097 } }
I20260926 20:54:53.112073 21287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 889fb6342df04f799e959f47b74026bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "889fb6342df04f799e959f47b74026bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "889fb6342df04f799e959f47b74026bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43097 } } }
I20260926 20:54:53.112080 21286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 889fb6342df04f799e959f47b74026bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 889fb6342df04f799e959f47b74026bc. Latest consensus state: current_term: 1 leader_uuid: "889fb6342df04f799e959f47b74026bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "889fb6342df04f799e959f47b74026bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43097 } } }
I20260926 20:54:53.112223 21286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 889fb6342df04f799e959f47b74026bc [sys.catalog]: This master's current role is: LEADER
I20260926 20:54:53.112175 21287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 889fb6342df04f799e959f47b74026bc [sys.catalog]: This master's current role is: LEADER
I20260926 20:54:53.112535 21293 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:54:53.113076 21293 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:54:53.115916 21284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:53.108832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.173:59061 (local address 127.0.0.1:43097)
0926 20:54:53.109024 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:53.109035 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:53.109069 (+ 34us) server_negotiation.cc:407] Connection header received
0926 20:54:53.109893 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:53.109900 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:53.109954 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:53.110122 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:53.110442 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:53.112206 (+ 1764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:53.113633 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:53.114320 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:53.114443 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:53.115166 (+ 723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:53.115170 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:53.115170 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:54:53.115337 (+ 167us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:53.115367 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:54:53.115378 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:53.115454 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:53.115568 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 20:54:53.115656 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":73,"threads_started":1}
I20260926 20:54:53.116369 21293 catalog_manager.cc:1347] Generated new cluster ID: e5333c2178c547c2bcbe61dc840fed57
I20260926 20:54:53.116426 21293 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:54:53.116801 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:54:53.121759 21284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:53.118253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.173:47073 (local address 127.0.0.1:43097)
0926 20:54:53.118282 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:53.118283 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:53.118390 (+ 107us) server_negotiation.cc:407] Connection header received
0926 20:54:53.118447 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:53.118448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:53.118457 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:53.118513 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:53.118651 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:53.119358 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:53.120399 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:53.120701 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:53.120765 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:53.121488 (+ 723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:53.121489 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:53.121490 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:54:53.121494 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:53.121508 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:54:53.121512 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:53.121566 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:53.121662 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 20:54:53.121676 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260926 20:54:53.126715 21293 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:54:53.127194 21293 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:54:53.129676 21293 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 889fb6342df04f799e959f47b74026bc: Generated new TSK 0
I20260926 20:54:53.129863 21293 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:54:53.135862 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 21212
W20260926 20:54:53.139873 21197 connection.cc:535] client connection to 127.0.0.1:43097 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43097): Connection reset by peer (error 104)
2026-09-26T20:54:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1319 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455725523895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:54:53 dist-test-slave-hwlj krb5kdc[21311](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:54:53 dist-test-slave-hwlj krb5kdc[21311](info): set up 2 sockets
Sep 26 20:54:53 dist-test-slave-hwlj krb5kdc[21311](info): commencing operation
krb5kdc: starting...
W20260926 20:54:54.200007 392 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455725523895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455725523895-392-0/krb5kdc/test-user.keytab.
Sep 26 20:54:54 dist-test-slave-hwlj krb5kdc[21311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:54:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:54:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455725523895-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455725523895-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:54:54.318686 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46581
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455725523895-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46581
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:54:54.398090 21324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:54:54.398296 21324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:54:54.398319 21324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:54:54.399386 21324 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:54:54.399421 21324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:54:54.399435 21324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:54:54.399448 21324 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:54:54.399461 21324 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:54:54.400876 21324 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455725523895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46581
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455725523895-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:54:54.401058 21324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:54:54.401235 21324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:54:54.406078 21332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:54.406114 21329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:54.406280 21324 server_base.cc:1034] running on GCE node
W20260926 20:54:54.406334 21330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:54.406574 21324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:54:54.406916 21324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:54:54.408087 21324 hybrid_clock.cc:648] HybridClock initialized: now 1790456094408078 us; error 33 us; skew 500 ppm
Sep 26 20:54:54 dist-test-slave-hwlj krb5kdc[21311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:54:54.410538 21324 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:54:54.411021 21324 webserver.cc:460] Webserver started at http://127.0.0.1:45963/ using document root <none> and password file <none>
I20260926 20:54:54.411265 21324 fs_manager.cc:359] Metadata directory not provided
I20260926 20:54:54.411332 21324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:54:54.411478 21324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:54:54.412593 21324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455725523895-392-0/minicluster-data/master-0/data/instance:
uuid: "3edf005f062549ab9982cd4bbf6e32c8"
format_stamp: "Formatted at 2026-09-26 20:54:54 on dist-test-slave-hwlj"
server_key: "bcd73c51396133eb2386ccc8dade7c43"
server_key_iv: "2a7d7d5656745a91270d8926678b73dd"
server_key_version: "encryptionkey@0"
I20260926 20:54:54.412945 21324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455725523895-392-0/minicluster-data/master-0/wal/instance:
uuid: "3edf005f062549ab9982cd4bbf6e32c8"
format_stamp: "Formatted at 2026-09-26 20:54:54 on dist-test-slave-hwlj"
server_key: "bcd73c51396133eb2386ccc8dade7c43"
server_key_iv: "2a7d7d5656745a91270d8926678b73dd"
server_key_version: "encryptionkey@0"
I20260926 20:54:54.414232 21324 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 20:54:54.414975 21339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:54.415184 21324 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:54:54.415293 21324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "3edf005f062549ab9982cd4bbf6e32c8"
format_stamp: "Formatted at 2026-09-26 20:54:54 on dist-test-slave-hwlj"
server_key: "bcd73c51396133eb2386ccc8dade7c43"
server_key_iv: "2a7d7d5656745a91270d8926678b73dd"
server_key_version: "encryptionkey@0"
I20260926 20:54:54.415366 21324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:54:54.418668 21324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:54:54.420732 21324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:54:54.420885 21324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:54:54.424888 21324 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46581
I20260926 20:54:54.424917 21391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46581 every 8 connection(s)
I20260926 20:54:54.425261 21324 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:54:54.426203 21392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:54:54.428612 21392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3edf005f062549ab9982cd4bbf6e32c8: Bootstrap starting.
I20260926 20:54:54.429226 21392 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3edf005f062549ab9982cd4bbf6e32c8: Neither blocks nor log segments found. Creating new log.
I20260926 20:54:54.429507 21392 log.cc:824] T 00000000000000000000000000000000 P 3edf005f062549ab9982cd4bbf6e32c8: Log is configured to *not* fsync() on all Append() calls
I20260926 20:54:54.430243 21392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3edf005f062549ab9982cd4bbf6e32c8: No bootstrap required, opened a new log
I20260926 20:54:54.431886 21392 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3edf005f062549ab9982cd4bbf6e32c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3edf005f062549ab9982cd4bbf6e32c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46581 } }
I20260926 20:54:54.432019 21392 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3edf005f062549ab9982cd4bbf6e32c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:54:54.432044 21392 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3edf005f062549ab9982cd4bbf6e32c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3edf005f062549ab9982cd4bbf6e32c8, State: Initialized, Role: FOLLOWER
I20260926 20:54:54.432219 21392 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3edf005f062549ab9982cd4bbf6e32c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3edf005f062549ab9982cd4bbf6e32c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46581 } }
I20260926 20:54:54.432283 21392 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3edf005f062549ab9982cd4bbf6e32c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:54:54.432304 21392 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3edf005f062549ab9982cd4bbf6e32c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:54:54.432371 21392 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3edf005f062549ab9982cd4bbf6e32c8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:54:54.432987 21392 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3edf005f062549ab9982cd4bbf6e32c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3edf005f062549ab9982cd4bbf6e32c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46581 } }
I20260926 20:54:54.433074 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 21324
I20260926 20:54:54.433101 21392 leader_election.cc:302] T 00000000000000000000000000000000 P 3edf005f062549ab9982cd4bbf6e32c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3edf005f062549ab9982cd4bbf6e32c8; no voters:
I20260926 20:54:54.433193 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455725523895-392-0/minicluster-data/master-0/wal/instance
I20260926 20:54:54.433292 21392 leader_election.cc:288] T 00000000000000000000000000000000 P 3edf005f062549ab9982cd4bbf6e32c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:54:54.433336 21395 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3edf005f062549ab9982cd4bbf6e32c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:54:54.433425 392 external_mini_cluster.cc:1428] Setting key 96fd167b134b19c109ace6e2f0f45669
I20260926 20:54:54.433493 21395 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3edf005f062549ab9982cd4bbf6e32c8 [term 1 LEADER]: Becoming Leader. State: Replica: 3edf005f062549ab9982cd4bbf6e32c8, State: Running, Role: LEADER
I20260926 20:54:54.433561 21392 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3edf005f062549ab9982cd4bbf6e32c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:54:54.433614 21395 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3edf005f062549ab9982cd4bbf6e32c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3edf005f062549ab9982cd4bbf6e32c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46581 } }
I20260926 20:54:54.434106 21397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3edf005f062549ab9982cd4bbf6e32c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3edf005f062549ab9982cd4bbf6e32c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3edf005f062549ab9982cd4bbf6e32c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46581 } } }
I20260926 20:54:54.433959 21395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3edf005f062549ab9982cd4bbf6e32c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3edf005f062549ab9982cd4bbf6e32c8. Latest consensus state: current_term: 1 leader_uuid: "3edf005f062549ab9982cd4bbf6e32c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3edf005f062549ab9982cd4bbf6e32c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46581 } } }
I20260926 20:54:54.434252 21397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3edf005f062549ab9982cd4bbf6e32c8 [sys.catalog]: This master's current role is: LEADER
I20260926 20:54:54.434450 21395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3edf005f062549ab9982cd4bbf6e32c8 [sys.catalog]: This master's current role is: LEADER
I20260926 20:54:54.434468 21406 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:54:54.435051 21406 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:54:54.436255 21402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:54.434065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.173:44331 (local address 127.0.0.1:46581)
0926 20:54:54.434597 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:54.434606 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:54.434638 (+ 32us) server_negotiation.cc:407] Connection header received
0926 20:54:54.434753 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:54.434760 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:54.434792 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:54.435206 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:54.435704 (+ 498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:54.435707 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:54.435708 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:54:54.435712 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:54.435819 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:54:54.435835 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:54.435925 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:54.436000 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 20:54:54.436053 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":89,"threads_started":1}
I20260926 20:54:54.436758 21406 catalog_manager.cc:1347] Generated new cluster ID: bbb5da3e8c84495f9fdb8cd0c9de480c
I20260926 20:54:54.436807 21406 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:54:54.437220 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:54:54.439814 21402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:54.438773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.173:55177 (local address 127.0.0.1:46581)
0926 20:54:54.438810 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:54.438812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:54.438926 (+ 114us) server_negotiation.cc:407] Connection header received
0926 20:54:54.438954 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:54.438955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:54.438961 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:54.439032 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:54.439578 (+ 546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:54.439580 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:54.439580 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:54:54.439584 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:54.439598 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:54:54.439603 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:54.439665 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:54.439719 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 20:54:54.439732 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 20:54:54.443229 21406 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:54:54.443393 21406 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:54:54.454025 21406 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3edf005f062549ab9982cd4bbf6e32c8: Generated new TSK 0
I20260926 20:54:54.454236 21406 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:54:54.457890 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 21324
2026-09-26T20:54:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1322 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455725523895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:54:54 dist-test-slave-hwlj krb5kdc[21423](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:54:54 dist-test-slave-hwlj krb5kdc[21423](info): set up 2 sockets
Sep 26 20:54:54 dist-test-slave-hwlj krb5kdc[21423](info): commencing operation
krb5kdc: starting...
W20260926 20:54:55.511999 392 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455725523895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455725523895-392-0/krb5kdc/test-user.keytab.
Sep 26 20:54:55 dist-test-slave-hwlj krb5kdc[21423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:54:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:54:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 20:54:55.628899 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36387
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:54:55.710062 21436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:54:55.710294 21436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:54:55.710319 21436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:54:55.711547 21436 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:54:55.711601 21436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:54:55.711615 21436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:54:55.711627 21436 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:54:55.711640 21436 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:54:55.713218 21436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455725523895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36387
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:54:55.713488 21436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:54:55.713730 21436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:54:55.719004 21442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:55.719015 21441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:55.719015 21444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:55.719280 21436 server_base.cc:1034] running on GCE node
I20260926 20:54:55.719534 21436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:54:55.719878 21436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:54:55.721055 21436 hybrid_clock.cc:648] HybridClock initialized: now 1790456095721042 us; error 33 us; skew 500 ppm
Sep 26 20:54:55 dist-test-slave-hwlj krb5kdc[21423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:54:55.723941 21436 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:54:55.724406 21436 webserver.cc:460] Webserver started at http://127.0.98.62:46549/ using document root <none> and password file <none>
I20260926 20:54:55.724663 21436 fs_manager.cc:359] Metadata directory not provided
I20260926 20:54:55.724740 21436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:54:55.724892 21436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:54:55.725973 21436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455725523895-392-0/minicluster-data/master-0/data/instance:
uuid: "f3f7d891b2384d41985d57be40116709"
format_stamp: "Formatted at 2026-09-26 20:54:55 on dist-test-slave-hwlj"
server_key: "5168cc3c68a8223c2cbb26b69c9e2c52"
server_key_iv: "5fc8c1adbb671ea5daaa8d17580651a8"
server_key_version: "encryptionkey@0"
I20260926 20:54:55.726356 21436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455725523895-392-0/minicluster-data/master-0/wal/instance:
uuid: "f3f7d891b2384d41985d57be40116709"
format_stamp: "Formatted at 2026-09-26 20:54:55 on dist-test-slave-hwlj"
server_key: "5168cc3c68a8223c2cbb26b69c9e2c52"
server_key_iv: "5fc8c1adbb671ea5daaa8d17580651a8"
server_key_version: "encryptionkey@0"
I20260926 20:54:55.727809 21436 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:54:55.728631 21451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:55.728817 21436 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:54:55.728920 21436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "f3f7d891b2384d41985d57be40116709"
format_stamp: "Formatted at 2026-09-26 20:54:55 on dist-test-slave-hwlj"
server_key: "5168cc3c68a8223c2cbb26b69c9e2c52"
server_key_iv: "5fc8c1adbb671ea5daaa8d17580651a8"
server_key_version: "encryptionkey@0"
I20260926 20:54:55.729004 21436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:54:55.750254 21436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:54:55.752741 21436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:54:55.752910 21436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:54:55.756963 21436 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36387
I20260926 20:54:55.757001 21503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36387 every 8 connection(s)
I20260926 20:54:55.757386 21436 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:54:55.758261 21504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:54:55.760598 21504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3f7d891b2384d41985d57be40116709: Bootstrap starting.
I20260926 20:54:55.761211 21504 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3f7d891b2384d41985d57be40116709: Neither blocks nor log segments found. Creating new log.
I20260926 20:54:55.761502 21504 log.cc:824] T 00000000000000000000000000000000 P f3f7d891b2384d41985d57be40116709: Log is configured to *not* fsync() on all Append() calls
I20260926 20:54:55.762213 21504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3f7d891b2384d41985d57be40116709: No bootstrap required, opened a new log
I20260926 20:54:55.763916 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 21436
I20260926 20:54:55.763872 21504 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3f7d891b2384d41985d57be40116709 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3f7d891b2384d41985d57be40116709" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36387 } }
I20260926 20:54:55.764029 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455725523895-392-0/minicluster-data/master-0/wal/instance
I20260926 20:54:55.764002 21504 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3f7d891b2384d41985d57be40116709 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:54:55.764065 21504 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3f7d891b2384d41985d57be40116709 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3f7d891b2384d41985d57be40116709, State: Initialized, Role: FOLLOWER
I20260926 20:54:55.764225 392 external_mini_cluster.cc:1428] Setting key 7b42e6164282081606910c9cb6b40678
I20260926 20:54:55.764240 21504 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3f7d891b2384d41985d57be40116709 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3f7d891b2384d41985d57be40116709" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36387 } }
I20260926 20:54:55.764339 21504 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3f7d891b2384d41985d57be40116709 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:54:55.764381 21504 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3f7d891b2384d41985d57be40116709 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:54:55.764423 21504 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3f7d891b2384d41985d57be40116709 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:54:55.765476 21504 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3f7d891b2384d41985d57be40116709 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3f7d891b2384d41985d57be40116709" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36387 } }
I20260926 20:54:55.765574 21504 leader_election.cc:302] T 00000000000000000000000000000000 P f3f7d891b2384d41985d57be40116709 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3f7d891b2384d41985d57be40116709; no voters:
I20260926 20:54:55.765744 21504 leader_election.cc:288] T 00000000000000000000000000000000 P f3f7d891b2384d41985d57be40116709 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:54:55.765793 21509 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3f7d891b2384d41985d57be40116709 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:54:55.765985 21509 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3f7d891b2384d41985d57be40116709 [term 1 LEADER]: Becoming Leader. State: Replica: f3f7d891b2384d41985d57be40116709, State: Running, Role: LEADER
I20260926 20:54:55.766031 21504 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3f7d891b2384d41985d57be40116709 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:54:55.767267 21509 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3f7d891b2384d41985d57be40116709 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3f7d891b2384d41985d57be40116709" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36387 } }
I20260926 20:54:55.768528 21511 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3f7d891b2384d41985d57be40116709 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3f7d891b2384d41985d57be40116709" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3f7d891b2384d41985d57be40116709" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36387 } } }
I20260926 20:54:55.768627 21511 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3f7d891b2384d41985d57be40116709 [sys.catalog]: This master's current role is: LEADER
I20260926 20:54:55.768903 21510 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3f7d891b2384d41985d57be40116709 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3f7d891b2384d41985d57be40116709. Latest consensus state: current_term: 1 leader_uuid: "f3f7d891b2384d41985d57be40116709" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3f7d891b2384d41985d57be40116709" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36387 } } }
I20260926 20:54:55.768970 21510 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3f7d891b2384d41985d57be40116709 [sys.catalog]: This master's current role is: LEADER
W20260926 20:54:55.769101 21524 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f3f7d891b2384d41985d57be40116709: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:54:55.769202 21524 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:54:55.769276 21525 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:54:55.769416 21525 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:54:55.770783 21525 catalog_manager.cc:1347] Generated new cluster ID: 51e9c0d291384ec79343110243485098
I20260926 20:54:55.770838 21525 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 20:54:55 dist-test-slave-hwlj krb5kdc[21423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:54:55.775537 21508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:55.764762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44777 (local address 127.0.98.62:36387)
0926 20:54:55.764964 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:55.764972 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:55.764997 (+ 25us) server_negotiation.cc:407] Connection header received
0926 20:54:55.765101 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:55.765106 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:55.765146 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:55.765280 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:55.766094 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:55.767077 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:55.768265 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:55.768926 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:55.769042 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:55.771908 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:55.771912 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:55.771913 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:54:55.771925 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:55.773855 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:55.774311 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:55.774314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:55.774314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:55.774428 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:55.774670 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:55.774671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:55.774671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:55.774963 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:55.775114 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:55.775241 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 20:54:55.775335 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":94,"threads_started":1}
I20260926 20:54:55.776525 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:54:55.785137 21508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:55.778050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55761 (local address 127.0.98.62:36387)
0926 20:54:55.778098 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:55.778099 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:55.778107 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:54:55.778207 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:55.778209 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:55.778219 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:55.778279 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:55.778858 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:55.779609 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:55.780614 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:55.780931 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:55.781003 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:55.782351 (+ 1348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:55.782353 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:55.782353 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:54:55.782360 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:55.783959 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:55.784355 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:55.784356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:55.784356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:55.784414 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:55.784642 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:55.784643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:55.784643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:55.784775 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:55.784873 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:55.784985 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 20:54:55.785046 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 20:54:55.803115 21525 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:54:55.803695 21525 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:54:55.809845 21525 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3f7d891b2384d41985d57be40116709: Generated new TSK 0
I20260926 20:54:55.810048 21525 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:54:55.814289 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 21436
W20260926 20:54:55.818488 21421 connection.cc:535] client connection to 127.0.98.62:36387 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:36387): Connection reset by peer (error 104)
2026-09-26T20:54:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1356 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455725523895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:54:55 dist-test-slave-hwlj krb5kdc[21535](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:54:55 dist-test-slave-hwlj krb5kdc[21535](info): set up 2 sockets
Sep 26 20:54:55 dist-test-slave-hwlj krb5kdc[21535](info): commencing operation
krb5kdc: starting...
W20260926 20:54:56.843896 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455725523895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455725523895-392-0/krb5kdc/test-user.keytab.
Sep 26 20:54:56 dist-test-slave-hwlj krb5kdc[21535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:54:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:54:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 20:54:56.963075 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40777
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:54:57.042879 21548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:54:57.043121 21548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:54:57.043159 21548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:54:57.044305 21548 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:54:57.044374 21548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:54:57.044400 21548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:54:57.044427 21548 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:54:57.044459 21548 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:54:57.045922 21548 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455725523895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40777
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:54:57.046195 21548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:54:57.046415 21548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:54:57.051399 21556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:57.051442 21554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:57.051453 21548 server_base.cc:1034] running on GCE node
W20260926 20:54:57.051455 21553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:57.051868 21548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:54:57.052188 21548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:54:57.053349 21548 hybrid_clock.cc:648] HybridClock initialized: now 1790456097053338 us; error 32 us; skew 500 ppm
Sep 26 20:54:57 dist-test-slave-hwlj krb5kdc[21535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456097, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:54:57.062294 21548 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:54:57.062790 21548 webserver.cc:460] Webserver started at http://127.0.98.62:44933/ using document root <none> and password file <none>
I20260926 20:54:57.063032 21548 fs_manager.cc:359] Metadata directory not provided
I20260926 20:54:57.063107 21548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:54:57.063261 21548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:54:57.064419 21548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455725523895-392-0/minicluster-data/master-0/data/instance:
uuid: "30fe521b7f954c6da95561ca1b819e6e"
format_stamp: "Formatted at 2026-09-26 20:54:57 on dist-test-slave-hwlj"
server_key: "35f205b5a84446e4062d5122761931be"
server_key_iv: "a27c6a8904c72fabc97c92e995c7005f"
server_key_version: "encryptionkey@0"
I20260926 20:54:57.064805 21548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455725523895-392-0/minicluster-data/master-0/wal/instance:
uuid: "30fe521b7f954c6da95561ca1b819e6e"
format_stamp: "Formatted at 2026-09-26 20:54:57 on dist-test-slave-hwlj"
server_key: "35f205b5a84446e4062d5122761931be"
server_key_iv: "a27c6a8904c72fabc97c92e995c7005f"
server_key_version: "encryptionkey@0"
I20260926 20:54:57.066147 21548 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 20:54:57.066920 21563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:57.067097 21548 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 20:54:57.067211 21548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "30fe521b7f954c6da95561ca1b819e6e"
format_stamp: "Formatted at 2026-09-26 20:54:57 on dist-test-slave-hwlj"
server_key: "35f205b5a84446e4062d5122761931be"
server_key_iv: "a27c6a8904c72fabc97c92e995c7005f"
server_key_version: "encryptionkey@0"
I20260926 20:54:57.067294 21548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:54:57.088799 21548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:54:57.090626 21548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:54:57.090749 21548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:54:57.094681 21548 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40777
I20260926 20:54:57.094751 21615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40777 every 8 connection(s)
I20260926 20:54:57.095081 21548 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:54:57.096014 21616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:54:57.097927 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 21548
I20260926 20:54:57.098018 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455725523895-392-0/minicluster-data/master-0/wal/instance
I20260926 20:54:57.098224 392 external_mini_cluster.cc:1428] Setting key 1fd82f9f826e6cce2c077b085c331b94
I20260926 20:54:57.098397 21616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30fe521b7f954c6da95561ca1b819e6e: Bootstrap starting.
I20260926 20:54:57.099143 21616 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30fe521b7f954c6da95561ca1b819e6e: Neither blocks nor log segments found. Creating new log.
I20260926 20:54:57.099426 21616 log.cc:824] T 00000000000000000000000000000000 P 30fe521b7f954c6da95561ca1b819e6e: Log is configured to *not* fsync() on all Append() calls
I20260926 20:54:57.100198 21616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30fe521b7f954c6da95561ca1b819e6e: No bootstrap required, opened a new log
I20260926 20:54:57.102344 21616 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30fe521b7f954c6da95561ca1b819e6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30fe521b7f954c6da95561ca1b819e6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40777 } }
I20260926 20:54:57.102511 21616 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30fe521b7f954c6da95561ca1b819e6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:54:57.102550 21616 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30fe521b7f954c6da95561ca1b819e6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30fe521b7f954c6da95561ca1b819e6e, State: Initialized, Role: FOLLOWER
I20260926 20:54:57.102732 21616 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30fe521b7f954c6da95561ca1b819e6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30fe521b7f954c6da95561ca1b819e6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40777 } }
I20260926 20:54:57.102856 21616 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30fe521b7f954c6da95561ca1b819e6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:54:57.102903 21616 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30fe521b7f954c6da95561ca1b819e6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:54:57.102958 21616 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30fe521b7f954c6da95561ca1b819e6e [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:54:57.103657 21616 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30fe521b7f954c6da95561ca1b819e6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30fe521b7f954c6da95561ca1b819e6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40777 } }
I20260926 20:54:57.103783 21616 leader_election.cc:302] T 00000000000000000000000000000000 P 30fe521b7f954c6da95561ca1b819e6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30fe521b7f954c6da95561ca1b819e6e; no voters:
I20260926 20:54:57.103960 21619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:57.098737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53641 (local address 127.0.98.62:40777)
0926 20:54:57.098924 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:57.098932 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:57.098956 (+ 24us) server_negotiation.cc:407] Connection header received
0926 20:54:57.099433 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:57.099440 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:57.099487 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:57.099664 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:57.099941 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:57.100885 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:57.102026 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:57.102410 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:57.102485 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:57.103278 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:57.103282 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:57.103282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:54:57.103441 (+ 159us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:57.103470 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:54:57.103480 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:57.103542 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:57.103672 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 20:54:57.103722 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":98,"threads_started":1}
I20260926 20:54:57.103971 21616 leader_election.cc:288] T 00000000000000000000000000000000 P 30fe521b7f954c6da95561ca1b819e6e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:54:57.104000 21621 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30fe521b7f954c6da95561ca1b819e6e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:54:57.104146 21621 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30fe521b7f954c6da95561ca1b819e6e [term 1 LEADER]: Becoming Leader. State: Replica: 30fe521b7f954c6da95561ca1b819e6e, State: Running, Role: LEADER
I20260926 20:54:57.104239 21621 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30fe521b7f954c6da95561ca1b819e6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30fe521b7f954c6da95561ca1b819e6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40777 } }
I20260926 20:54:57.104266 21616 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30fe521b7f954c6da95561ca1b819e6e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:54:57.104588 21622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30fe521b7f954c6da95561ca1b819e6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30fe521b7f954c6da95561ca1b819e6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30fe521b7f954c6da95561ca1b819e6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40777 } } }
I20260926 20:54:57.104605 21623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30fe521b7f954c6da95561ca1b819e6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30fe521b7f954c6da95561ca1b819e6e. Latest consensus state: current_term: 1 leader_uuid: "30fe521b7f954c6da95561ca1b819e6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30fe521b7f954c6da95561ca1b819e6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40777 } } }
I20260926 20:54:57.104697 21622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30fe521b7f954c6da95561ca1b819e6e [sys.catalog]: This master's current role is: LEADER
I20260926 20:54:57.104750 21623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30fe521b7f954c6da95561ca1b819e6e [sys.catalog]: This master's current role is: LEADER
I20260926 20:54:57.105026 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:54:57.105105 21625 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:54:57.105607 21625 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:54:57.107270 21625 catalog_manager.cc:1347] Generated new cluster ID: 22f6a356966e4e7bb602517e9937ba32
I20260926 20:54:57.107321 21625 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:54:57.110323 21619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:57.106584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33253 (local address 127.0.98.62:40777)
0926 20:54:57.106619 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:57.106621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:57.106832 (+ 211us) server_negotiation.cc:407] Connection header received
0926 20:54:57.106935 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:57.106936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:57.106945 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:57.106972 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:57.107220 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:57.107954 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:57.108981 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:57.109312 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:57.109369 (+ 57us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:57.110062 (+ 693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:57.110064 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:57.110064 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:54:57.110069 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:57.110083 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:54:57.110087 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:57.110142 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:57.110246 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 20:54:57.110255 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 20:54:57.120298 21625 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:54:57.121028 21625 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:54:57.128199 21625 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30fe521b7f954c6da95561ca1b819e6e: Generated new TSK 0
I20260926 20:54:57.128394 21625 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:54:57.134608 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 21548
W20260926 20:54:57.140694 21533 connection.cc:535] client connection to 127.0.98.62:40777 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:40777): Connection reset by peer (error 104)
2026-09-26T20:54:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1322 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455725523895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:54:57 dist-test-slave-hwlj krb5kdc[21647](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:54:57 dist-test-slave-hwlj krb5kdc[21647](info): set up 2 sockets
Sep 26 20:54:57 dist-test-slave-hwlj krb5kdc[21647](info): commencing operation
krb5kdc: starting...
W20260926 20:54:58.180011 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455725523895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455725523895-392-0/krb5kdc/test-user.keytab.
Sep 26 20:54:58 dist-test-slave-hwlj krb5kdc[21647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:54:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:54:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 20:54:58.303970 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35821
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:54:58.385211 21660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:54:58.385495 21660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:54:58.385541 21660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:54:58.386780 21660 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:54:58.386860 21660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:54:58.386884 21660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:54:58.386914 21660 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:54:58.386945 21660 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:54:58.388516 21660 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455725523895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35821
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:54:58.388826 21660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:54:58.389073 21660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:54:58.394040 21668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:58.394070 21666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:58.394380 21660 server_base.cc:1034] running on GCE node
W20260926 20:54:58.394040 21665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:58.394706 21660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:54:58.395061 21660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:54:58.396268 21660 hybrid_clock.cc:648] HybridClock initialized: now 1790456098396272 us; error 48 us; skew 500 ppm
Sep 26 20:54:58 dist-test-slave-hwlj krb5kdc[21647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:54:58.398936 21660 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:54:58.399505 21660 webserver.cc:460] Webserver started at http://127.0.98.62:34317/ using document root <none> and password file <none>
I20260926 20:54:58.399813 21660 fs_manager.cc:359] Metadata directory not provided
I20260926 20:54:58.399891 21660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:54:58.400034 21660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:54:58.401125 21660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455725523895-392-0/minicluster-data/master-0/data/instance:
uuid: "ec062eb23bb94700ab63b7ce729f28f8"
format_stamp: "Formatted at 2026-09-26 20:54:58 on dist-test-slave-hwlj"
server_key: "6b4cfbbdba5880b5970f56251a619e31"
server_key_iv: "c5b3717a306b64b9abe329532769cfc9"
server_key_version: "encryptionkey@0"
I20260926 20:54:58.401500 21660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455725523895-392-0/minicluster-data/master-0/wal/instance:
uuid: "ec062eb23bb94700ab63b7ce729f28f8"
format_stamp: "Formatted at 2026-09-26 20:54:58 on dist-test-slave-hwlj"
server_key: "6b4cfbbdba5880b5970f56251a619e31"
server_key_iv: "c5b3717a306b64b9abe329532769cfc9"
server_key_version: "encryptionkey@0"
I20260926 20:54:58.402891 21660 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:54:58.403664 21675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:58.403865 21660 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:54:58.403967 21660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "ec062eb23bb94700ab63b7ce729f28f8"
format_stamp: "Formatted at 2026-09-26 20:54:58 on dist-test-slave-hwlj"
server_key: "6b4cfbbdba5880b5970f56251a619e31"
server_key_iv: "c5b3717a306b64b9abe329532769cfc9"
server_key_version: "encryptionkey@0"
I20260926 20:54:58.404057 21660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:54:58.407343 21660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:54:58.409679 21660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:54:58.409852 21660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:54:58.413631 21660 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35821
I20260926 20:54:58.413658 21727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35821 every 8 connection(s)
I20260926 20:54:58.414004 21660 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:54:58.415133 21728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:54:58.417801 21728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec062eb23bb94700ab63b7ce729f28f8: Bootstrap starting.
I20260926 20:54:58.418444 21728 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec062eb23bb94700ab63b7ce729f28f8: Neither blocks nor log segments found. Creating new log.
I20260926 20:54:58.418853 21728 log.cc:824] T 00000000000000000000000000000000 P ec062eb23bb94700ab63b7ce729f28f8: Log is configured to *not* fsync() on all Append() calls
I20260926 20:54:58.419392 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 21660
I20260926 20:54:58.419507 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455725523895-392-0/minicluster-data/master-0/wal/instance
I20260926 20:54:58.419708 392 external_mini_cluster.cc:1428] Setting key 4166d1979072aa9fbd257c0f304bb41b
I20260926 20:54:58.419723 21728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec062eb23bb94700ab63b7ce729f28f8: No bootstrap required, opened a new log
I20260926 20:54:58.421905 21728 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec062eb23bb94700ab63b7ce729f28f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec062eb23bb94700ab63b7ce729f28f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35821 } }
I20260926 20:54:58.422071 21728 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec062eb23bb94700ab63b7ce729f28f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:54:58.422159 21728 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec062eb23bb94700ab63b7ce729f28f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec062eb23bb94700ab63b7ce729f28f8, State: Initialized, Role: FOLLOWER
I20260926 20:54:58.422349 21728 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec062eb23bb94700ab63b7ce729f28f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec062eb23bb94700ab63b7ce729f28f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35821 } }
I20260926 20:54:58.422461 21728 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec062eb23bb94700ab63b7ce729f28f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:54:58.422508 21728 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec062eb23bb94700ab63b7ce729f28f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:54:58.422557 21728 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec062eb23bb94700ab63b7ce729f28f8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:54:58.422700 21732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:58.420274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41437 (local address 127.0.98.62:35821)
0926 20:54:58.420499 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:58.420512 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:58.420541 (+ 29us) server_negotiation.cc:407] Connection header received
0926 20:54:58.421018 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:58.421025 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:58.421051 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:58.421204 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:58.421942 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:58.421951 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:58.421951 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:54:58.421955 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:58.421991 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:54:58.422002 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:58.422078 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:58.422147 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 20:54:58.422305 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":97,"threads_started":1}
I20260926 20:54:58.423454 21728 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec062eb23bb94700ab63b7ce729f28f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec062eb23bb94700ab63b7ce729f28f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35821 } }
I20260926 20:54:58.423588 21728 leader_election.cc:302] T 00000000000000000000000000000000 P ec062eb23bb94700ab63b7ce729f28f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec062eb23bb94700ab63b7ce729f28f8; no voters:
I20260926 20:54:58.423595 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:54:58.423799 21728 leader_election.cc:288] T 00000000000000000000000000000000 P ec062eb23bb94700ab63b7ce729f28f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:54:58.423851 21733 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec062eb23bb94700ab63b7ce729f28f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:54:58.424033 21733 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec062eb23bb94700ab63b7ce729f28f8 [term 1 LEADER]: Becoming Leader. State: Replica: ec062eb23bb94700ab63b7ce729f28f8, State: Running, Role: LEADER
I20260926 20:54:58.424069 21728 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec062eb23bb94700ab63b7ce729f28f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:54:58.425549 21733 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec062eb23bb94700ab63b7ce729f28f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec062eb23bb94700ab63b7ce729f28f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35821 } }
I20260926 20:54:58.426206 21734 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec062eb23bb94700ab63b7ce729f28f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec062eb23bb94700ab63b7ce729f28f8. Latest consensus state: current_term: 1 leader_uuid: "ec062eb23bb94700ab63b7ce729f28f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec062eb23bb94700ab63b7ce729f28f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35821 } } }
I20260926 20:54:58.426349 21735 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec062eb23bb94700ab63b7ce729f28f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec062eb23bb94700ab63b7ce729f28f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec062eb23bb94700ab63b7ce729f28f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35821 } } }
I20260926 20:54:58.426445 21734 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec062eb23bb94700ab63b7ce729f28f8 [sys.catalog]: This master's current role is: LEADER
I20260926 20:54:58.426448 21735 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec062eb23bb94700ab63b7ce729f28f8 [sys.catalog]: This master's current role is: LEADER
I20260926 20:54:58.426561 21732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:58.425252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38835 (local address 127.0.98.62:35821)
0926 20:54:58.425292 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:58.425294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:58.425447 (+ 153us) server_negotiation.cc:407] Connection header received
0926 20:54:58.425521 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:58.425521 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:58.425527 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:58.425600 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:58.426300 (+ 700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:58.426301 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:58.426302 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:54:58.426307 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:58.426321 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:54:58.426326 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:58.426358 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:58.426482 (+ 124us) server_negotiation.cc:299] Negotiation successful
0926 20:54:58.426496 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 20:54:58.427137 21752 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ec062eb23bb94700ab63b7ce729f28f8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:54:58.427198 21752 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:54:58.427269 21753 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:54:58.427419 21753 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:54:58.428889 21753 catalog_manager.cc:1347] Generated new cluster ID: 2275f6318cf44454b970a6e462b3d3cb
I20260926 20:54:58.428951 21753 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:54:58.454595 21753 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:54:58.454811 21753 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:54:58.461304 21753 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec062eb23bb94700ab63b7ce729f28f8: Generated new TSK 0
I20260926 20:54:58.461539 21753 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:54:58.473237 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 21660
2026-09-26T20:54:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1337 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455725523895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:54:58 dist-test-slave-hwlj krb5kdc[21759](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:54:58 dist-test-slave-hwlj krb5kdc[21759](info): set up 2 sockets
Sep 26 20:54:58 dist-test-slave-hwlj krb5kdc[21759](info): commencing operation
krb5kdc: starting...
W20260926 20:54:59.503710 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455725523895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455725523895-392-0/krb5kdc/test-user.keytab.
Sep 26 20:54:59 dist-test-slave-hwlj krb5kdc[21759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:54:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:54:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 20:54:59.622634 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45451
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:54:59.704162 21772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:54:59.704370 21772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:54:59.704396 21772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:54:59.705974 21772 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:54:59.706034 21772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:54:59.706050 21772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:54:59.706063 21772 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:54:59.706075 21772 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:54:59.707796 21772 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455725523895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:54:59.708055 21772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:54:59.708326 21772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:54:59.713475 21777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:59.713490 21778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:59.713474 21780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:59.713802 21772 server_base.cc:1034] running on GCE node
I20260926 20:54:59.714073 21772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:54:59.714386 21772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:54:59.715595 21772 hybrid_clock.cc:648] HybridClock initialized: now 1790456099715579 us; error 40 us; skew 500 ppm
Sep 26 20:54:59 dist-test-slave-hwlj krb5kdc[21759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:54:59.718360 21772 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:54:59.718766 21772 webserver.cc:460] Webserver started at http://127.0.98.62:41187/ using document root <none> and password file <none>
I20260926 20:54:59.718966 21772 fs_manager.cc:359] Metadata directory not provided
I20260926 20:54:59.719005 21772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:54:59.719103 21772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:54:59.720213 21772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455725523895-392-0/minicluster-data/master-0/data/instance:
uuid: "047c3095447243f0a4f208b1fae2a880"
format_stamp: "Formatted at 2026-09-26 20:54:59 on dist-test-slave-hwlj"
server_key: "094464bc3e233ada1014097e3fb8ce7d"
server_key_iv: "07c7ea7a625637dd6a4ee61f7ff09049"
server_key_version: "encryptionkey@0"
I20260926 20:54:59.720515 21772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455725523895-392-0/minicluster-data/master-0/wal/instance:
uuid: "047c3095447243f0a4f208b1fae2a880"
format_stamp: "Formatted at 2026-09-26 20:54:59 on dist-test-slave-hwlj"
server_key: "094464bc3e233ada1014097e3fb8ce7d"
server_key_iv: "07c7ea7a625637dd6a4ee61f7ff09049"
server_key_version: "encryptionkey@0"
I20260926 20:54:59.721834 21772 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:54:59.722594 21787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:59.722816 21772 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:54:59.722913 21772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "047c3095447243f0a4f208b1fae2a880"
format_stamp: "Formatted at 2026-09-26 20:54:59 on dist-test-slave-hwlj"
server_key: "094464bc3e233ada1014097e3fb8ce7d"
server_key_iv: "07c7ea7a625637dd6a4ee61f7ff09049"
server_key_version: "encryptionkey@0"
I20260926 20:54:59.722987 21772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:54:59.740989 21772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:54:59.743018 21772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:54:59.743132 21772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:54:59.746966 21772 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45451
I20260926 20:54:59.747078 21839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45451 every 8 connection(s)
I20260926 20:54:59.747417 21772 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:54:59.748457 21840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:54:59.750660 21840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 047c3095447243f0a4f208b1fae2a880: Bootstrap starting.
I20260926 20:54:59.751199 21840 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 047c3095447243f0a4f208b1fae2a880: Neither blocks nor log segments found. Creating new log.
I20260926 20:54:59.751425 21840 log.cc:824] T 00000000000000000000000000000000 P 047c3095447243f0a4f208b1fae2a880: Log is configured to *not* fsync() on all Append() calls
I20260926 20:54:59.752130 21840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 047c3095447243f0a4f208b1fae2a880: No bootstrap required, opened a new log
I20260926 20:54:59.753687 21840 raft_consensus.cc:348] T 00000000000000000000000000000000 P 047c3095447243f0a4f208b1fae2a880 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "047c3095447243f0a4f208b1fae2a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45451 } }
I20260926 20:54:59.753825 21840 raft_consensus.cc:374] T 00000000000000000000000000000000 P 047c3095447243f0a4f208b1fae2a880 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:54:59.753849 21840 raft_consensus.cc:729] T 00000000000000000000000000000000 P 047c3095447243f0a4f208b1fae2a880 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 047c3095447243f0a4f208b1fae2a880, State: Initialized, Role: FOLLOWER
I20260926 20:54:59.753949 21840 consensus_queue.cc:260] T 00000000000000000000000000000000 P 047c3095447243f0a4f208b1fae2a880 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "047c3095447243f0a4f208b1fae2a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45451 } }
I20260926 20:54:59.753997 21840 raft_consensus.cc:388] T 00000000000000000000000000000000 P 047c3095447243f0a4f208b1fae2a880 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:54:59.754017 21840 raft_consensus.cc:482] T 00000000000000000000000000000000 P 047c3095447243f0a4f208b1fae2a880 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:54:59.754043 21840 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 047c3095447243f0a4f208b1fae2a880 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:54:59.754498 21840 raft_consensus.cc:504] T 00000000000000000000000000000000 P 047c3095447243f0a4f208b1fae2a880 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "047c3095447243f0a4f208b1fae2a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45451 } }
I20260926 20:54:59.754585 21840 leader_election.cc:302] T 00000000000000000000000000000000 P 047c3095447243f0a4f208b1fae2a880 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 047c3095447243f0a4f208b1fae2a880; no voters:
I20260926 20:54:59.754738 21840 leader_election.cc:288] T 00000000000000000000000000000000 P 047c3095447243f0a4f208b1fae2a880 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:54:59.754776 21843 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 047c3095447243f0a4f208b1fae2a880 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:54:59.754899 21843 raft_consensus.cc:686] T 00000000000000000000000000000000 P 047c3095447243f0a4f208b1fae2a880 [term 1 LEADER]: Becoming Leader. State: Replica: 047c3095447243f0a4f208b1fae2a880, State: Running, Role: LEADER
I20260926 20:54:59.754999 21840 sys_catalog.cc:564] T 00000000000000000000000000000000 P 047c3095447243f0a4f208b1fae2a880 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:54:59.755061 21843 consensus_queue.cc:237] T 00000000000000000000000000000000 P 047c3095447243f0a4f208b1fae2a880 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "047c3095447243f0a4f208b1fae2a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45451 } }
I20260926 20:54:59.755512 21844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 047c3095447243f0a4f208b1fae2a880 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "047c3095447243f0a4f208b1fae2a880" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "047c3095447243f0a4f208b1fae2a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45451 } } }
I20260926 20:54:59.755529 21845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 047c3095447243f0a4f208b1fae2a880 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 047c3095447243f0a4f208b1fae2a880. Latest consensus state: current_term: 1 leader_uuid: "047c3095447243f0a4f208b1fae2a880" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "047c3095447243f0a4f208b1fae2a880" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45451 } } }
I20260926 20:54:59.755642 21844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 047c3095447243f0a4f208b1fae2a880 [sys.catalog]: This master's current role is: LEADER
I20260926 20:54:59.755861 21845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 047c3095447243f0a4f208b1fae2a880 [sys.catalog]: This master's current role is: LEADER
I20260926 20:54:59.756104 21850 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:54:59.756639 21850 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:54:59.757529 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 21772
I20260926 20:54:59.757632 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455725523895-392-0/minicluster-data/master-0/wal/instance
I20260926 20:54:59.757838 392 external_mini_cluster.cc:1428] Setting key 236e4e96140910f03a3e23541592e457
I20260926 20:54:59.758150 21850 catalog_manager.cc:1347] Generated new cluster ID: d82befc18efe4c7ea933175f85893c63
I20260926 20:54:59.758205 21850 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 20:54:59 dist-test-slave-hwlj krb5kdc[21759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:54:59.767848 21863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:59.758276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42421 (local address 127.0.98.62:45451)
0926 20:54:59.758427 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:59.758438 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:59.758636 (+ 198us) server_negotiation.cc:407] Connection header received
0926 20:54:59.758809 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:59.758823 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:59.758869 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:59.759005 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:59.759656 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:59.760659 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:59.761469 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:59.761900 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:59.762037 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:59.764407 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:59.764411 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:59.764412 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:54:59.764424 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:59.766181 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:59.766589 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:59.766592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:59.766593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:59.766692 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:59.766919 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:59.766920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:59.766921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:59.767163 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:59.767323 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:59.767422 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 20:54:59.767560 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":70,"threads_started":1}
I20260926 20:54:59.768730 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:54:59.775667 21850 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:54:59.776455 21850 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:54:59.776469 21863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:59.770149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38943 (local address 127.0.98.62:45451)
0926 20:54:59.770183 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:59.770185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:59.770304 (+ 119us) server_negotiation.cc:407] Connection header received
0926 20:54:59.770372 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:59.770373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:59.770384 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:59.770438 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:59.770939 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:59.771447 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:59.772259 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:59.772484 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:59.772574 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:59.773735 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:59.773736 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:59.773736 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:54:59.773743 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:59.775170 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:59.775594 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:59.775595 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:59.775596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:59.775659 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:59.775900 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:59.775900 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:59.775901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:59.776019 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:59.776139 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:59.776252 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 20:54:59.776383 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":17}
I20260926 20:54:59.782616 21850 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 047c3095447243f0a4f208b1fae2a880: Generated new TSK 0
I20260926 20:54:59.782805 21850 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:54:59.786499 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 21772
2026-09-26T20:54:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1312 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455725523895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:54:59 dist-test-slave-hwlj krb5kdc[21871](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:54:59 dist-test-slave-hwlj krb5kdc[21871](info): set up 2 sockets
Sep 26 20:54:59 dist-test-slave-hwlj krb5kdc[21871](info): commencing operation
krb5kdc: starting...
W20260926 20:55:00.832041 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455725523895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455725523895-392-0/krb5kdc/test-user.keytab.
Sep 26 20:55:00 dist-test-slave-hwlj krb5kdc[21871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:55:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:55:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 20:55:00.955996 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40189
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:55:01.040757 21884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:55:01.040968 21884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:55:01.041002 21884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:55:01.042209 21884 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:55:01.042258 21884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:55:01.042273 21884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:55:01.042286 21884 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:55:01.042299 21884 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:55:01.043898 21884 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455725523895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:55:01.044168 21884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:55:01.044406 21884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:55:01.049525 21892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:01.049520 21889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:01.049530 21890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:01.049729 21884 server_base.cc:1034] running on GCE node
I20260926 20:55:01.050160 21884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:55:01.050532 21884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:55:01.051712 21884 hybrid_clock.cc:648] HybridClock initialized: now 1790456101051705 us; error 26 us; skew 500 ppm
Sep 26 20:55:01 dist-test-slave-hwlj krb5kdc[21871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456101, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:55:01.054488 21884 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:55:01.055038 21884 webserver.cc:460] Webserver started at http://127.0.98.62:35293/ using document root <none> and password file <none>
I20260926 20:55:01.055328 21884 fs_manager.cc:359] Metadata directory not provided
I20260926 20:55:01.055402 21884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:55:01.055564 21884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:55:01.056804 21884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455725523895-392-0/minicluster-data/master-0/data/instance:
uuid: "03d6b343b6164d4f81df54801b6439b1"
format_stamp: "Formatted at 2026-09-26 20:55:01 on dist-test-slave-hwlj"
server_key: "6d7930f6fdff6d36346dba0a7a57ee33"
server_key_iv: "a9ce3670893194e2dffafa3410bdbe99"
server_key_version: "encryptionkey@0"
I20260926 20:55:01.057194 21884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455725523895-392-0/minicluster-data/master-0/wal/instance:
uuid: "03d6b343b6164d4f81df54801b6439b1"
format_stamp: "Formatted at 2026-09-26 20:55:01 on dist-test-slave-hwlj"
server_key: "6d7930f6fdff6d36346dba0a7a57ee33"
server_key_iv: "a9ce3670893194e2dffafa3410bdbe99"
server_key_version: "encryptionkey@0"
I20260926 20:55:01.058645 21884 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:55:01.059463 21899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:01.059855 21884 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:55:01.059988 21884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "03d6b343b6164d4f81df54801b6439b1"
format_stamp: "Formatted at 2026-09-26 20:55:01 on dist-test-slave-hwlj"
server_key: "6d7930f6fdff6d36346dba0a7a57ee33"
server_key_iv: "a9ce3670893194e2dffafa3410bdbe99"
server_key_version: "encryptionkey@0"
I20260926 20:55:01.060067 21884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:55:01.069798 21884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:55:01.072012 21884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:55:01.072188 21884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:55:01.076156 21884 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40189
I20260926 20:55:01.076184 21951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40189 every 8 connection(s)
I20260926 20:55:01.076536 21884 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:55:01.077405 21952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:55:01.079680 21952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03d6b343b6164d4f81df54801b6439b1: Bootstrap starting.
I20260926 20:55:01.080273 21952 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03d6b343b6164d4f81df54801b6439b1: Neither blocks nor log segments found. Creating new log.
I20260926 20:55:01.080561 21952 log.cc:824] T 00000000000000000000000000000000 P 03d6b343b6164d4f81df54801b6439b1: Log is configured to *not* fsync() on all Append() calls
I20260926 20:55:01.081292 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 21884
I20260926 20:55:01.081406 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455725523895-392-0/minicluster-data/master-0/wal/instance
I20260926 20:55:01.081413 21952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03d6b343b6164d4f81df54801b6439b1: No bootstrap required, opened a new log
I20260926 20:55:01.081653 392 external_mini_cluster.cc:1428] Setting key 47531adcd7d5471c1e479020507dc419
I20260926 20:55:01.083240 21952 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03d6b343b6164d4f81df54801b6439b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03d6b343b6164d4f81df54801b6439b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40189 } }
I20260926 20:55:01.083401 21952 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03d6b343b6164d4f81df54801b6439b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:55:01.083465 21952 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03d6b343b6164d4f81df54801b6439b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03d6b343b6164d4f81df54801b6439b1, State: Initialized, Role: FOLLOWER
I20260926 20:55:01.083652 21952 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03d6b343b6164d4f81df54801b6439b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03d6b343b6164d4f81df54801b6439b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40189 } }
I20260926 20:55:01.083740 21952 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03d6b343b6164d4f81df54801b6439b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:55:01.083817 21952 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03d6b343b6164d4f81df54801b6439b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:55:01.083873 21952 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03d6b343b6164d4f81df54801b6439b1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:55:01.084641 21952 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03d6b343b6164d4f81df54801b6439b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03d6b343b6164d4f81df54801b6439b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40189 } }
I20260926 20:55:01.084746 21952 leader_election.cc:302] T 00000000000000000000000000000000 P 03d6b343b6164d4f81df54801b6439b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03d6b343b6164d4f81df54801b6439b1; no voters:
I20260926 20:55:01.084892 21952 leader_election.cc:288] T 00000000000000000000000000000000 P 03d6b343b6164d4f81df54801b6439b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:55:01.084957 21957 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03d6b343b6164d4f81df54801b6439b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:55:01.085134 21952 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03d6b343b6164d4f81df54801b6439b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:55:01.085137 21957 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03d6b343b6164d4f81df54801b6439b1 [term 1 LEADER]: Becoming Leader. State: Replica: 03d6b343b6164d4f81df54801b6439b1, State: Running, Role: LEADER
I20260926 20:55:01.085323 21957 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03d6b343b6164d4f81df54801b6439b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03d6b343b6164d4f81df54801b6439b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40189 } }
I20260926 20:55:01.085796 21958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03d6b343b6164d4f81df54801b6439b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03d6b343b6164d4f81df54801b6439b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03d6b343b6164d4f81df54801b6439b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40189 } } }
I20260926 20:55:01.086002 21958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03d6b343b6164d4f81df54801b6439b1 [sys.catalog]: This master's current role is: LEADER
I20260926 20:55:01.085943 21959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03d6b343b6164d4f81df54801b6439b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03d6b343b6164d4f81df54801b6439b1. Latest consensus state: current_term: 1 leader_uuid: "03d6b343b6164d4f81df54801b6439b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03d6b343b6164d4f81df54801b6439b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40189 } } }
I20260926 20:55:01.086391 21959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03d6b343b6164d4f81df54801b6439b1 [sys.catalog]: This master's current role is: LEADER
I20260926 20:55:01.086697 21972 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:55:01.087405 21972 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:55:01.089179 21972 catalog_manager.cc:1347] Generated new cluster ID: ea649f45cfba4f51b598e262f35578a9
I20260926 20:55:01.089223 21956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:01.082184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53173 (local address 127.0.98.62:40189)
0926 20:55:01.082371 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:01.082380 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:01.082413 (+ 33us) server_negotiation.cc:407] Connection header received
0926 20:55:01.083224 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:01.083230 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:01.083275 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:01.083419 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:01.083655 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:01.085399 (+ 1744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:01.086665 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:01.087317 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:01.087423 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:01.088420 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:01.088424 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:01.088425 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:55:01.088678 (+ 253us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:01.088725 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:55:01.088740 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:01.088820 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:01.088942 (+ 122us) server_negotiation.cc:299] Negotiation successful
0926 20:55:01.089013 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":81,"threads_started":1}
I20260926 20:55:01.089229 21972 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:55:01.090338 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:55:01.095559 21956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:01.092068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52001 (local address 127.0.98.62:40189)
0926 20:55:01.092100 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:01.092101 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:01.092109 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:55:01.092186 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:01.092187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:01.092196 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:01.092248 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:01.092384 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:01.093115 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:01.094199 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:01.094490 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:01.094555 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:01.095304 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:01.095305 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:01.095306 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:55:01.095311 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:01.095326 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:55:01.095330 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:01.095381 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:01.095451 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 20:55:01.095475 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260926 20:55:01.097961 21972 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:55:01.098495 21972 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:55:01.106012 21972 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03d6b343b6164d4f81df54801b6439b1: Generated new TSK 0
I20260926 20:55:01.106243 21972 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:55:01.109093 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 21884
2026-09-26T20:55:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1322 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455725523895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:55:01 dist-test-slave-hwlj krb5kdc[21983](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:55:01 dist-test-slave-hwlj krb5kdc[21983](info): set up 2 sockets
Sep 26 20:55:01 dist-test-slave-hwlj krb5kdc[21983](info): commencing operation
krb5kdc: starting...
W20260926 20:55:02.156087 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455725523895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455725523895-392-0/krb5kdc/test-user.keytab.
Sep 26 20:55:02 dist-test-slave-hwlj krb5kdc[21983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:55:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:55:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 20:55:02.287678 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43069
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:55:02.369994 21996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:55:02.370211 21996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:55:02.370234 21996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:55:02.371434 21996 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:55:02.371481 21996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:55:02.371495 21996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:55:02.371508 21996 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:55:02.371520 21996 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:55:02.373083 21996 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455725523895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:55:02.373355 21996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:55:02.373624 21996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:55:02.378837 22004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:02.378798 22001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:02.378818 22002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:02.379029 21996 server_base.cc:1034] running on GCE node
I20260926 20:55:02.379377 21996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:55:02.379701 21996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:55:02.380923 21996 hybrid_clock.cc:648] HybridClock initialized: now 1790456102380897 us; error 43 us; skew 500 ppm
Sep 26 20:55:02 dist-test-slave-hwlj krb5kdc[21983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:55:02.390146 21996 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:55:02.390653 21996 webserver.cc:460] Webserver started at http://127.0.98.62:33657/ using document root <none> and password file <none>
I20260926 20:55:02.390906 21996 fs_manager.cc:359] Metadata directory not provided
I20260926 20:55:02.390997 21996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:55:02.391149 21996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:55:02.392382 21996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455725523895-392-0/minicluster-data/master-0/data/instance:
uuid: "66f655cacf0843b6aaca046db8d26cca"
format_stamp: "Formatted at 2026-09-26 20:55:02 on dist-test-slave-hwlj"
server_key: "03ac34761bee7b2ab5787e43f1468282"
server_key_iv: "349aabcc3c88005e8b766fa1c9103a86"
server_key_version: "encryptionkey@0"
I20260926 20:55:02.392761 21996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455725523895-392-0/minicluster-data/master-0/wal/instance:
uuid: "66f655cacf0843b6aaca046db8d26cca"
format_stamp: "Formatted at 2026-09-26 20:55:02 on dist-test-slave-hwlj"
server_key: "03ac34761bee7b2ab5787e43f1468282"
server_key_iv: "349aabcc3c88005e8b766fa1c9103a86"
server_key_version: "encryptionkey@0"
I20260926 20:55:02.394134 21996 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:55:02.394985 22011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:02.395166 21996 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:55:02.395273 21996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "66f655cacf0843b6aaca046db8d26cca"
format_stamp: "Formatted at 2026-09-26 20:55:02 on dist-test-slave-hwlj"
server_key: "03ac34761bee7b2ab5787e43f1468282"
server_key_iv: "349aabcc3c88005e8b766fa1c9103a86"
server_key_version: "encryptionkey@0"
I20260926 20:55:02.395357 21996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:55:02.398729 21996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:55:02.401067 21996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:55:02.401233 21996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:55:02.405165 21996 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43069
I20260926 20:55:02.405200 22063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43069 every 8 connection(s)
I20260926 20:55:02.405568 21996 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:55:02.407004 22064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:55:02.409397 22064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66f655cacf0843b6aaca046db8d26cca: Bootstrap starting.
I20260926 20:55:02.409969 22064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66f655cacf0843b6aaca046db8d26cca: Neither blocks nor log segments found. Creating new log.
I20260926 20:55:02.410251 22064 log.cc:824] T 00000000000000000000000000000000 P 66f655cacf0843b6aaca046db8d26cca: Log is configured to *not* fsync() on all Append() calls
I20260926 20:55:02.410943 22064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66f655cacf0843b6aaca046db8d26cca: No bootstrap required, opened a new log
I20260926 20:55:02.412525 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 21996
I20260926 20:55:02.412678 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455725523895-392-0/minicluster-data/master-0/wal/instance
I20260926 20:55:02.412680 22064 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66f655cacf0843b6aaca046db8d26cca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66f655cacf0843b6aaca046db8d26cca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43069 } }
I20260926 20:55:02.412868 22064 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66f655cacf0843b6aaca046db8d26cca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:55:02.412897 392 external_mini_cluster.cc:1428] Setting key 29861e5c31c451009f525469db6ca8a8
I20260926 20:55:02.412911 22064 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66f655cacf0843b6aaca046db8d26cca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66f655cacf0843b6aaca046db8d26cca, State: Initialized, Role: FOLLOWER
I20260926 20:55:02.413103 22064 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66f655cacf0843b6aaca046db8d26cca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66f655cacf0843b6aaca046db8d26cca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43069 } }
I20260926 20:55:02.413196 22064 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66f655cacf0843b6aaca046db8d26cca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:55:02.413233 22064 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66f655cacf0843b6aaca046db8d26cca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:55:02.413293 22064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66f655cacf0843b6aaca046db8d26cca [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:55:02.414487 22064 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66f655cacf0843b6aaca046db8d26cca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66f655cacf0843b6aaca046db8d26cca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43069 } }
I20260926 20:55:02.414623 22064 leader_election.cc:302] T 00000000000000000000000000000000 P 66f655cacf0843b6aaca046db8d26cca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66f655cacf0843b6aaca046db8d26cca; no voters:
I20260926 20:55:02.414804 22064 leader_election.cc:288] T 00000000000000000000000000000000 P 66f655cacf0843b6aaca046db8d26cca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:55:02.414845 22069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66f655cacf0843b6aaca046db8d26cca [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:55:02.415006 22069 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66f655cacf0843b6aaca046db8d26cca [term 1 LEADER]: Becoming Leader. State: Replica: 66f655cacf0843b6aaca046db8d26cca, State: Running, Role: LEADER
I20260926 20:55:02.415098 22064 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66f655cacf0843b6aaca046db8d26cca [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:55:02.415148 22069 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66f655cacf0843b6aaca046db8d26cca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66f655cacf0843b6aaca046db8d26cca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43069 } }
I20260926 20:55:02.415400 22068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:02.413494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60243 (local address 127.0.98.62:43069)
0926 20:55:02.413767 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:02.413778 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:02.413820 (+ 42us) server_negotiation.cc:407] Connection header received
0926 20:55:02.413963 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:02.413973 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:02.414026 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:02.414172 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:02.414880 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:02.414885 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:02.414885 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:55:02.414890 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:02.414940 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:55:02.414957 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:02.415030 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:02.415150 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 20:55:02.415197 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":155,"threads_started":1}
I20260926 20:55:02.415606 22069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66f655cacf0843b6aaca046db8d26cca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66f655cacf0843b6aaca046db8d26cca. Latest consensus state: current_term: 1 leader_uuid: "66f655cacf0843b6aaca046db8d26cca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66f655cacf0843b6aaca046db8d26cca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43069 } } }
I20260926 20:55:02.415720 22069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66f655cacf0843b6aaca046db8d26cca [sys.catalog]: This master's current role is: LEADER
I20260926 20:55:02.415616 22070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66f655cacf0843b6aaca046db8d26cca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66f655cacf0843b6aaca046db8d26cca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66f655cacf0843b6aaca046db8d26cca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43069 } } }
I20260926 20:55:02.415846 22070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66f655cacf0843b6aaca046db8d26cca [sys.catalog]: This master's current role is: LEADER
I20260926 20:55:02.416175 22078 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:55:02.416661 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:55:02.416774 22078 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:55:02.418255 22078 catalog_manager.cc:1347] Generated new cluster ID: 3fa9c0b7fe204092b1372f434363cb34
I20260926 20:55:02.418329 22078 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:55:02.419526 22068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:02.418334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59683 (local address 127.0.98.62:43069)
0926 20:55:02.418367 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:02.418369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:02.418542 (+ 173us) server_negotiation.cc:407] Connection header received
0926 20:55:02.418607 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:02.418608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:02.418614 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:02.418660 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:02.419293 (+ 633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:02.419295 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:02.419295 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:55:02.419299 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:02.419313 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:55:02.419317 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:02.419364 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:02.419429 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 20:55:02.419443 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 20:55:02.441363 22078 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:55:02.441582 22078 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:55:02.447847 22078 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66f655cacf0843b6aaca046db8d26cca: Generated new TSK 0
I20260926 20:55:02.448068 22078 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:55:02.452957 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 21996
2026-09-26T20:55:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1344 ms)
[----------] 18 tests from AuthTokenIssuingTest (23842 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:55:02 dist-test-slave-hwlj krb5kdc[22095](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:55:02 dist-test-slave-hwlj krb5kdc[22095](info): set up 2 sockets
Sep 26 20:55:02 dist-test-slave-hwlj krb5kdc[22095](info): commencing operation
krb5kdc: starting...
W20260926 20:55:03.508034 392 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/krb5kdc/test-user.keytab.
Sep 26 20:55:03 dist-test-slave-hwlj krb5kdc[22095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:55:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:55:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 20:55:03.634377 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46149
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46149,127.0.98.61:40001,127.0.98.60:45851
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:55:03.716562 22108 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 20:55:03.716797 22108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:55:03.716815 22108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:55:03.716840 22108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:55:03.717967 22108 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:55:03.718003 22108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:55:03.718017 22108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:55:03.718030 22108 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:55:03.718042 22108 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:55:03.719475 22108 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46149,127.0.98.61:40001,127.0.98.60:45851
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:55:03.719672 22108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:55:03.719933 22108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:55:03.724817 22114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:03.724807 22116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:03.724807 22113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:03.724905 22108 server_base.cc:1034] running on GCE node
I20260926 20:55:03.725334 22108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:55:03.725670 22108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:55:03.726853 22108 hybrid_clock.cc:648] HybridClock initialized: now 1790456103726825 us; error 37 us; skew 500 ppm
Sep 26 20:55:03 dist-test-slave-hwlj krb5kdc[22095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:55:03.729506 22108 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:55:03.730017 22108 webserver.cc:460] Webserver started at http://127.0.98.62:45955/ using document root <none> and password file <none>
I20260926 20:55:03.730245 22108 fs_manager.cc:359] Metadata directory not provided
I20260926 20:55:03.730314 22108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:55:03.730468 22108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:55:03.731529 22108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/minicluster-data/master-0/data/instance:
uuid: "8375fe14c9b64fd3a8750b09e4932d6a"
format_stamp: "Formatted at 2026-09-26 20:55:03 on dist-test-slave-hwlj"
server_key: "f33e5a0a405418095c6f1503f9fd8397"
server_key_iv: "bed33fa4bda12f65c82edf588eab9ce9"
server_key_version: "encryptionkey@0"
I20260926 20:55:03.731933 22108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/minicluster-data/master-0/wal/instance:
uuid: "8375fe14c9b64fd3a8750b09e4932d6a"
format_stamp: "Formatted at 2026-09-26 20:55:03 on dist-test-slave-hwlj"
server_key: "f33e5a0a405418095c6f1503f9fd8397"
server_key_iv: "bed33fa4bda12f65c82edf588eab9ce9"
server_key_version: "encryptionkey@0"
I20260926 20:55:03.733316 22108 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:55:03.734203 22123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:03.734452 22108 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:55:03.734562 22108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "8375fe14c9b64fd3a8750b09e4932d6a"
format_stamp: "Formatted at 2026-09-26 20:55:03 on dist-test-slave-hwlj"
server_key: "f33e5a0a405418095c6f1503f9fd8397"
server_key_iv: "bed33fa4bda12f65c82edf588eab9ce9"
server_key_version: "encryptionkey@0"
I20260926 20:55:03.734664 22108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:55:03.748488 22108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:55:03.750559 22108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:55:03.750733 22108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:55:03.754822 22108 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46149
I20260926 20:55:03.754860 22175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46149 every 8 connection(s)
I20260926 20:55:03.755257 22108 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:55:03.756242 22176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:55:03.757790 22176 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46149 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:55:03.759455 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 22108
I20260926 20:55:03.759577 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/minicluster-data/master-0/wal/instance
I20260926 20:55:03.759816 392 external_mini_cluster.cc:1428] Setting key d91470206a7e322376453f29d3d7a9bd
Sep 26 20:55:03 dist-test-slave-hwlj krb5kdc[22095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260926 20:55:03.768719 22179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:03.758512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58268 (local address 127.0.98.62:46149)
0926 20:55:03.758700 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:03.758709 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:03.758833 (+ 124us) server_negotiation.cc:407] Connection header received
0926 20:55:03.759397 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:03.759399 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:03.759425 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:03.759521 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:03.760181 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:03.761342 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:03.762364 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:03.762884 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:03.763005 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:03.765190 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:03.765197 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:03.765197 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:03.765204 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:03.767131 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:03.767502 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:03.767505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:03.767506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:03.767606 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:03.767882 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:03.767883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:03.767883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:03.768195 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:03.768279 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:03.768533 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 20:55:03.768603 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":88,"threads_started":1}
I20260926 20:55:03.768749 22178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:03.758485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46149 (local address 127.0.0.1:58268)
0926 20:55:03.758691 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:03.758734 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:03.759242 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:03.759525 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:03.759528 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:03.759880 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:55:03.760082 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:03.760087 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:03.761467 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:03.761468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:03.762274 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:03.762277 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:03.763020 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:03.763022 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:03.763081 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:03.763615 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:55:03.763623 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:55:03.765100 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:55:03.767224 (+ 2124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:03.767230 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:03.767231 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:03.767415 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:03.767676 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:03.767677 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:03.767677 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:03.767815 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:03.768272 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:55:03.768278 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:55:03.768336 (+ 58us) client_negotiation.cc:769] Sending connection context
0926 20:55:03.768406 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 20:55:03.768522 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":85,"threads_started":1}
I20260926 20:55:03.769502 22176 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40001 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:55:03.769858 22178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:03.769748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40001 (local address 127.0.0.1:42522)
0926 20:55:03.769799 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:03.769839 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 20:55:03.769999 22124 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40001: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260926 20:55:03.770449 22178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:03.770344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40001 (local address 127.0.0.1:42524)
0926 20:55:03.770390 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:03.770420 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 20:55:03.770548 22176 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40001: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40001: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260926 20:55:03.784793 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40001
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46149,127.0.98.61:40001,127.0.98.60:45851
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:55:03.819881 22176 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40001 } attempt: 1
I20260926 20:55:03.820382 22178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:03.820232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40001 (local address 127.0.0.1:42526)
0926 20:55:03.820302 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:03.820346 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 20:55:03.820771 22178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:03.820668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40001 (local address 127.0.0.1:42530)
0926 20:55:03.820716 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:03.820744 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 20:55:03.820861 22176 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40001: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40001: connect: Connection refused (error 111)
W20260926 20:55:03.865850 22183 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 20:55:03.866070 22183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:55:03.866087 22183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:55:03.866111 22183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:55:03.867648 22183 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:55:03.867699 22183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:55:03.867714 22183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:55:03.867727 22183 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:55:03.867739 22183 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:55:03.869316 22183 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46149,127.0.98.61:40001,127.0.98.60:45851
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:55:03.869509 22183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:55:03.869690 22183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:55:03.874640 22191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:03.874652 22188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:03.874686 22183 server_base.cc:1034] running on GCE node
W20260926 20:55:03.874688 22189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:03.875186 22183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:55:03.875509 22183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:55:03.876681 22183 hybrid_clock.cc:648] HybridClock initialized: now 1790456103876674 us; error 54 us; skew 500 ppm
Sep 26 20:55:03 dist-test-slave-hwlj krb5kdc[22095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:55:03.879151 22183 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260926 20:55:03.879609 22183 webserver.cc:460] Webserver started at http://127.0.98.61:33635/ using document root <none> and password file <none>
I20260926 20:55:03.879907 22183 fs_manager.cc:359] Metadata directory not provided
I20260926 20:55:03.879987 22183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:55:03.880144 22183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:55:03.881213 22183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/minicluster-data/master-1/data/instance:
uuid: "8dd574607f5d47debbb98db6c49e002a"
format_stamp: "Formatted at 2026-09-26 20:55:03 on dist-test-slave-hwlj"
server_key: "b3e92bfde4a274d9f81ec0c1cb81073e"
server_key_iv: "403713b06c15acbde478d066c0dd1cef"
server_key_version: "encryptionkey@0"
I20260926 20:55:03.881587 22183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/minicluster-data/master-1/wal/instance:
uuid: "8dd574607f5d47debbb98db6c49e002a"
format_stamp: "Formatted at 2026-09-26 20:55:03 on dist-test-slave-hwlj"
server_key: "b3e92bfde4a274d9f81ec0c1cb81073e"
server_key_iv: "403713b06c15acbde478d066c0dd1cef"
server_key_version: "encryptionkey@0"
I20260926 20:55:03.882944 22183 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:55:03.883739 22198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:03.884038 22183 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:55:03.884153 22183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/minicluster-data/master-1/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/minicluster-data/master-1/wal
uuid: "8dd574607f5d47debbb98db6c49e002a"
format_stamp: "Formatted at 2026-09-26 20:55:03 on dist-test-slave-hwlj"
server_key: "b3e92bfde4a274d9f81ec0c1cb81073e"
server_key_iv: "403713b06c15acbde478d066c0dd1cef"
server_key_version: "encryptionkey@0"
I20260926 20:55:03.884239 22183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:55:03.889001 22176 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40001 } attempt: 2
I20260926 20:55:03.889408 22178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:03.889287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40001 (local address 127.0.0.1:42538)
0926 20:55:03.889341 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:03.889379 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 20:55:03.889730 22178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:03.889649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40001 (local address 127.0.0.1:42544)
0926 20:55:03.889690 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:03.889714 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 20:55:03.889846 22176 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40001: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40001: connect: Connection refused (error 111)
I20260926 20:55:03.898125 22183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:55:03.900743 22183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:55:03.900930 22183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:55:03.905287 22183 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40001
I20260926 20:55:03.905315 22250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40001 every 8 connection(s)
I20260926 20:55:03.905720 22183 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/minicluster-data/master-1/data/info.pb
I20260926 20:55:03.906751 22251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:55:03.908414 22251 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46149 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:55:03.909515 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 22183
I20260926 20:55:03.909626 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/minicluster-data/master-1/wal/instance
I20260926 20:55:03.909843 392 external_mini_cluster.cc:1428] Setting key 99c301d7ce885ef3d234eaebe1ab2d14
Sep 26 20:55:03 dist-test-slave-hwlj krb5kdc[22095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260926 20:55:03.918812 22179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:03.909143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58280 (local address 127.0.98.62:46149)
0926 20:55:03.909188 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:03.909190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:03.909482 (+ 292us) server_negotiation.cc:407] Connection header received
0926 20:55:03.910173 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:03.910174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:03.910188 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:03.910247 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:03.911103 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:03.911622 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:03.912773 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:03.912998 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:03.913083 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:03.915599 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:03.915601 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:03.915601 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:03.915610 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:03.917294 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:03.917750 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:03.917751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:03.917752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:03.917813 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:03.918093 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:03.918094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:03.918094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:03.918226 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:03.918305 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:03.918614 (+ 309us) server_negotiation.cc:299] Negotiation successful
0926 20:55:03.918706 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 20:55:03.919045 22253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:03.909123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46149 (local address 127.0.0.1:58280)
0926 20:55:03.909332 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:03.909379 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:03.910062 (+ 683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:03.910321 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:03.910325 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:03.910721 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:55:03.911006 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:03.911012 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:03.911740 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:03.911742 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:03.912648 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:03.912651 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:03.913105 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:03.913107 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:03.913251 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:03.913826 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:55:03.913835 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:55:03.915490 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:55:03.917395 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:03.917402 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:03.917405 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:03.917654 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:03.917887 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:03.917889 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:03.917889 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:03.918010 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:03.918307 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:55:03.918315 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:55:03.918468 (+ 153us) client_negotiation.cc:769] Sending connection context
0926 20:55:03.918579 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 20:55:03.918764 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":116,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260926 20:55:03.919710 22251 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40001 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:55:03 dist-test-slave-hwlj krb5kdc[22095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 20:55:03.928755 22253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:03.920019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40001 (local address 127.0.0.1:42558)
0926 20:55:03.920073 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:03.920085 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:03.920126 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:03.920395 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:03.920396 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:03.920542 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:55:03.920588 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:03.920590 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:03.921915 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:03.921916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:03.922913 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:03.922916 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:03.923404 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:03.923405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:03.923470 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:03.924043 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:55:03.924047 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:55:03.925431 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:55:03.927319 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:03.927320 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:03.927321 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:03.927571 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:03.927906 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:03.927907 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:03.927907 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:03.927987 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:03.928360 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:55:03.928361 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:55:03.928417 (+ 56us) client_negotiation.cc:769] Sending connection context
0926 20:55:03.928507 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 20:55:03.928633 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 20:55:03.928856 22255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:03.920093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42558 (local address 127.0.98.61:40001)
0926 20:55:03.920249 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:03.920254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:03.920272 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:55:03.920290 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:03.920292 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:03.920311 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:03.920396 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:03.920691 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:03.921823 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:03.923005 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:03.923317 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:03.923408 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:03.925528 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:03.925536 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:03.925536 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:03.925543 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:03.927217 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:03.927688 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:03.927691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:03.927692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:03.927818 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:03.928093 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:03.928094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:03.928094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:03.928262 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:03.928338 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:03.928561 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 20:55:03.928688 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":130,"thread_start_us":104,"threads_started":1}
I20260926 20:55:03.929530 22251 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45851 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:55:03.929943 22253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:03.929814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45851 (local address 127.0.0.1:52578)
0926 20:55:03.929867 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:03.929922 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 20:55:03.930083 22199 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45851: connect: Connection refused (error 111)
I20260926 20:55:03.930538 22253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:03.930432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45851 (local address 127.0.0.1:52590)
0926 20:55:03.930486 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:03.930514 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 20:55:03.930675 22251 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45851: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45851: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260926 20:55:03.940850 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45851
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46149,127.0.98.61:40001,127.0.98.60:45851
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:55:03.979935 22251 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45851 } attempt: 1
I20260926 20:55:03.980504 22253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:03.980348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45851 (local address 127.0.0.1:52594)
0926 20:55:03.980413 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:03.980463 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 20:55:03.980952 22253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:03.980853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45851 (local address 127.0.0.1:52600)
0926 20:55:03.980902 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:03.980932 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 20:55:03.981055 22176 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40001 } attempt: 3
W20260926 20:55:03.981091 22251 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45851: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45851: connect: Connection refused (error 111)
Sep 26 20:55:03 dist-test-slave-hwlj krb5kdc[22095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 20:55:03.988327 22178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:03.981359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40001 (local address 127.0.0.1:42572)
0926 20:55:03.981408 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:03.981421 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:03.981470 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:03.981661 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:03.981662 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:03.981848 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:55:03.981921 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:03.981923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:03.982598 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:03.982600 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:03.983128 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:03.983130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:03.983521 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:03.983522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:03.983607 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:03.984125 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:55:03.984130 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:55:03.985364 (+ 1234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:55:03.987143 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:03.987145 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:03.987145 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:03.987326 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:03.987534 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:03.987535 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:03.987536 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:03.987605 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:03.987962 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:55:03.987963 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:55:03.988021 (+ 58us) client_negotiation.cc:769] Sending connection context
0926 20:55:03.988113 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 20:55:03.988224 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 20:55:03.988327 22255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:03.981408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42572 (local address 127.0.98.61:40001)
0926 20:55:03.981462 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:03.981464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:03.981474 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:55:03.981540 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:03.981542 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:03.981553 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:03.981652 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:03.982005 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:03.982502 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:03.983216 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:03.983433 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:03.983520 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:03.985456 (+ 1936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:03.985458 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:03.985458 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:03.985466 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:03.987041 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:03.987404 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:03.987405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:03.987405 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:03.987464 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:03.987696 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:03.987697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:03.987697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:03.987906 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:03.987970 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:03.988140 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 20:55:03.988225 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 20:55:03.988750 22176 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45851 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:55:03.989110 22178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:03.989007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45851 (local address 127.0.0.1:52616)
0926 20:55:03.989040 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:03.989087 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260926 20:55:03.989405 22178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:03.989351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45851 (local address 127.0.0.1:52630)
0926 20:55:03.989373 (+ 22us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:03.989393 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260926 20:55:03.989506 22176 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45851: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45851: connect: Connection refused (error 111)
I20260926 20:55:04.020661 22176 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45851 } attempt: 1
I20260926 20:55:04.021178 22178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:04.021036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45851 (local address 127.0.0.1:52632)
0926 20:55:04.021108 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:04.021152 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 20:55:04.021561 22178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:04.021482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45851 (local address 127.0.0.1:52644)
0926 20:55:04.021522 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:04.021546 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 20:55:04.021667 22176 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45851: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45851: connect: Connection refused (error 111)
W20260926 20:55:04.030901 22258 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 20:55:04.031128 22258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:55:04.031147 22258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:55:04.031168 22258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:55:04.032351 22258 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:55:04.032394 22258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:55:04.032408 22258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:55:04.032421 22258 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:55:04.032433 22258 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:55:04.033871 22258 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46149,127.0.98.61:40001,127.0.98.60:45851
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:55:04.034096 22258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:55:04.034283 22258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:55:04.039324 22258 server_base.cc:1034] running on GCE node
W20260926 20:55:04.039306 22266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:04.039306 22264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:04.039294 22263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:04.039892 22258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:55:04.040161 22258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:55:04.041303 22258 hybrid_clock.cc:648] HybridClock initialized: now 1790456104041291 us; error 27 us; skew 500 ppm
Sep 26 20:55:04 dist-test-slave-hwlj krb5kdc[22095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456104, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:55:04.043955 22258 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260926 20:55:04.044395 22258 webserver.cc:460] Webserver started at http://127.0.98.60:43189/ using document root <none> and password file <none>
I20260926 20:55:04.044655 22258 fs_manager.cc:359] Metadata directory not provided
I20260926 20:55:04.044699 22258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:55:04.044863 22258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:55:04.045948 22258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/minicluster-data/master-2/data/instance:
uuid: "b99d668819104d0b96de636d6b88a991"
format_stamp: "Formatted at 2026-09-26 20:55:04 on dist-test-slave-hwlj"
server_key: "de7641d8d78af8c448940ff6859cb807"
server_key_iv: "a7201728c471b3616ae3ea4593413cc8"
server_key_version: "encryptionkey@0"
I20260926 20:55:04.046312 22258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/minicluster-data/master-2/wal/instance:
uuid: "b99d668819104d0b96de636d6b88a991"
format_stamp: "Formatted at 2026-09-26 20:55:04 on dist-test-slave-hwlj"
server_key: "de7641d8d78af8c448940ff6859cb807"
server_key_iv: "a7201728c471b3616ae3ea4593413cc8"
server_key_version: "encryptionkey@0"
I20260926 20:55:04.047631 22258 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 20:55:04.048391 22273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:04.048600 22258 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:55:04.048692 22258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/minicluster-data/master-2/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/minicluster-data/master-2/wal
uuid: "b99d668819104d0b96de636d6b88a991"
format_stamp: "Formatted at 2026-09-26 20:55:04 on dist-test-slave-hwlj"
server_key: "de7641d8d78af8c448940ff6859cb807"
server_key_iv: "a7201728c471b3616ae3ea4593413cc8"
server_key_version: "encryptionkey@0"
I20260926 20:55:04.048769 22258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:55:04.049254 22251 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45851 } attempt: 2
I20260926 20:55:04.049626 22253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:04.049483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45851 (local address 127.0.0.1:52648)
0926 20:55:04.049549 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:04.049591 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 20:55:04.049980 22253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:04.049886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45851 (local address 127.0.0.1:52662)
0926 20:55:04.049934 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:04.049961 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 20:55:04.050081 22251 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45851: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45851: connect: Connection refused (error 111)
I20260926 20:55:04.065640 22258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:55:04.068192 22258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:55:04.068324 22258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:55:04.072270 22258 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:45851
I20260926 20:55:04.072396 22325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:45851 every 8 connection(s)
I20260926 20:55:04.072700 22258 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/minicluster-data/master-2/data/info.pb
I20260926 20:55:04.073752 22326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:55:04.075325 22326 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46149 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:55:04.076417 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 22258
I20260926 20:55:04.076565 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455725523895-392-0/minicluster-data/master-2/wal/instance
I20260926 20:55:04.076787 392 external_mini_cluster.cc:1428] Setting key f45c6bf2fda0d2ee62be25dcafb6922d
Sep 26 20:55:04 dist-test-slave-hwlj krb5kdc[22095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456104, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 26 20:55:04 dist-test-slave-hwlj krb5kdc[22095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 20:55:04.085985 22179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:04.076098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58284 (local address 127.0.98.62:46149)
0926 20:55:04.076150 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:04.076152 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:04.076382 (+ 230us) server_negotiation.cc:407] Connection header received
0926 20:55:04.077187 (+ 805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:04.077189 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:04.077204 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:04.077276 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:04.077928 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:04.078420 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:04.079355 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:04.079555 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:04.079646 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:04.082280 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:04.082282 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:04.082282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:04.082290 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:04.084300 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:04.084769 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:04.084771 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:04.084771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:04.084839 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:04.085215 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:04.085216 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:04.085216 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:04.085360 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:04.085433 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:04.085786 (+ 353us) server_negotiation.cc:299] Negotiation successful
0926 20:55:04.085871 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 20:55:04.086113 22328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:04.076079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46149 (local address 127.0.0.1:58284)
0926 20:55:04.076262 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:04.076297 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:04.076999 (+ 702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:04.077355 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:04.077359 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:04.077626 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:55:04.077830 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:04.077834 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:04.078508 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:04.078509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:04.079258 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:04.079260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:04.079685 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:04.079686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:04.079886 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:04.080440 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:55:04.080449 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:55:04.082185 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:55:04.084408 (+ 2223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:04.084414 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:04.084416 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:04.084689 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:04.084943 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:04.084944 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:04.084945 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:04.085134 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:04.085436 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:55:04.085441 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:55:04.085633 (+ 192us) client_negotiation.cc:769] Sending connection context
0926 20:55:04.085738 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 20:55:04.085865 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":87,"threads_started":1}
I20260926 20:55:04.086714 22330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:04.077175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52664 (local address 127.0.98.60:45851)
0926 20:55:04.077354 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:04.077357 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:04.077504 (+ 147us) server_negotiation.cc:407] Connection header received
0926 20:55:04.077562 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:04.077565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:04.077590 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:04.077681 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:04.078289 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:04.079100 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:04.079915 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:04.080224 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:04.080298 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:04.082936 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:04.082945 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:04.082946 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:04.082954 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:04.084955 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:04.085372 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:04.085375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:04.085376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:04.085437 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:04.085648 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:04.085648 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:04.085649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:04.085876 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:04.086049 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:04.086484 (+ 435us) server_negotiation.cc:299] Negotiation successful
0926 20:55:04.086593 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":80,"threads_started":1}
I20260926 20:55:04.087019 22326 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40001 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:55:04 dist-test-slave-hwlj krb5kdc[22095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456104, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 26 20:55:04 dist-test-slave-hwlj krb5kdc[22095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:55:04.095350 22328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:04.087338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40001 (local address 127.0.0.1:42574)
0926 20:55:04.087479 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:04.087491 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:04.087541 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:04.087668 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:04.087669 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:04.087877 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:55:04.087922 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:04.087923 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:04.088615 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:04.088615 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:04.089195 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:04.089197 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:04.089556 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:04.089557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:04.089625 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:04.090140 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:55:04.090145 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:55:04.091645 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:55:04.094201 (+ 2556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:04.094202 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:04.094203 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:04.094383 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:04.094629 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:04.094630 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:04.094630 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:04.094690 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:04.094983 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:55:04.094984 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:55:04.095047 (+ 63us) client_negotiation.cc:769] Sending connection context
0926 20:55:04.095131 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 20:55:04.095245 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125}
I20260926 20:55:04.095350 22255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:04.087431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42574 (local address 127.0.98.61:40001)
0926 20:55:04.087487 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:04.087489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:04.087529 (+ 40us) server_negotiation.cc:407] Connection header received
0926 20:55:04.087595 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:04.087596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:04.087607 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:04.087665 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:04.087986 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:04.088528 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:04.089289 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:04.089487 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:04.089540 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:04.091812 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:04.091814 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:04.091814 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:04.091823 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:04.094128 (+ 2305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:04.094464 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:04.094466 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:04.094466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:04.094530 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:04.094762 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:04.094763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:04.094764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:04.094926 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:04.094989 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:04.095171 (+ 182us) server_negotiation.cc:299] Negotiation successful
0926 20:55:04.095249 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 20:55:04.096223 22179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:04.087713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58290 (local address 127.0.98.62:46149)
0926 20:55:04.087771 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:04.087773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:04.087780 (+ 7us) server_negotiation.cc:407] Connection header received
0926 20:55:04.087798 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:04.087799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:04.087807 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:04.087915 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:04.088407 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:04.088921 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:04.089745 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:04.090038 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:04.090108 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:04.093160 (+ 3052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:04.093162 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:04.093162 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:04.093171 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:04.094842 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:04.095272 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:04.095273 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:04.095274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:04.095340 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:04.095607 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:04.095608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:04.095608 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:04.095791 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:04.095995 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:04.096035 (+ 40us) server_negotiation.cc:299] Negotiation successful
0926 20:55:04.096120 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 20:55:04.096352 22326 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45851 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:55:04.096804 22176 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45851 } attempt: 2
Sep 26 20:55:04 dist-test-slave-hwlj krb5kdc[22095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 26 20:55:04 dist-test-slave-hwlj krb5kdc[22095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456104, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 26 20:55:04 dist-test-slave-hwlj krb5kdc[22095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 20:55:04.105151 22178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:04.097041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45851 (local address 127.0.0.1:52680)
0926 20:55:04.097084 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:04.097094 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:04.097135 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:04.097371 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:04.097372 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:04.097524 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:55:04.097559 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:04.097560 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:04.098240 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:04.098241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:04.098735 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:04.098736 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:04.099222 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:04.099228 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:04.099293 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:04.099811 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:55:04.099817 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:55:04.101120 (+ 1303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:55:04.103854 (+ 2734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:04.103855 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:04.103856 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:04.104072 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:04.104301 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:04.104302 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:04.104302 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:04.104372 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:04.104754 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:55:04.104756 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:55:04.104807 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 20:55:04.104891 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 20:55:04.105034 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 20:55:04.105180 22331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:04.097071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52680 (local address 127.0.98.60:45851)
0926 20:55:04.097231 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:04.097233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:04.097240 (+ 7us) server_negotiation.cc:407] Connection header received
0926 20:55:04.097260 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:04.097261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:04.097270 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:04.097362 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:04.097638 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:04.098147 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:04.098820 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:04.099138 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:04.099204 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:04.101219 (+ 2015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:04.101221 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:04.101221 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:04.101230 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:04.103777 (+ 2547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:04.104159 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:04.104160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:04.104161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:04.104220 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:04.104460 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:04.104461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:04.104461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:04.104698 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:04.104762 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:04.104941 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 20:55:04.105048 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":136,"thread_start_us":99,"threads_started":1}
I20260926 20:55:04.105406 22330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:04.096710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52676 (local address 127.0.98.60:45851)
0926 20:55:04.096748 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:04.096750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:04.096758 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:55:04.096802 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:04.096803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:04.096815 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:04.096872 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:04.097308 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:04.098083 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:04.098986 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:04.099281 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:04.099371 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:04.101775 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:04.101777 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:04.101777 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:04.101784 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:04.104209 (+ 2425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:04.104593 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:04.104594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:04.104594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:04.104638 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:04.104852 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:04.104853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:04.104854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:04.104998 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:04.105052 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:04.105231 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 20:55:04.105306 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":19}
I20260926 20:55:04.105648 22328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:04.096635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45851 (local address 127.0.0.1:52676)
0926 20:55:04.096690 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:04.096703 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:04.096746 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:04.096889 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:04.096890 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:04.097174 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:55:04.097223 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:04.097226 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:04.098158 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:04.098159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:04.098916 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:04.098917 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:04.099374 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:04.099375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:04.099428 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:04.100108 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:55:04.100112 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:55:04.101681 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:55:04.104292 (+ 2611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:04.104293 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:04.104293 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:04.104520 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:04.104715 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:04.104716 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:04.104716 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:04.104780 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:04.105095 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:55:04.105096 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:55:04.105132 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 20:55:04.105474 (+ 342us) client_negotiation.cc:240] Negotiation successful
0926 20:55:04.105567 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 20:55:04.105741 22255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:04.097291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42584 (local address 127.0.98.61:40001)
0926 20:55:04.097327 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:04.097329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:04.097337 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:55:04.097357 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:04.097358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:04.097366 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:04.097411 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:04.098026 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:04.098745 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:04.099615 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:04.099970 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:04.100041 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:04.102635 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:04.102636 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:04.102636 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:04.102644 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:04.104342 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:04.104773 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:04.104774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:04.104775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:04.104831 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:04.105036 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:04.105037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:04.105037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:04.105166 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:04.105323 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:04.105556 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 20:55:04.105632 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 20:55:04.107069 22176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8375fe14c9b64fd3a8750b09e4932d6a: Bootstrap starting.
I20260926 20:55:04.107554 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:55:04.107929 22176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8375fe14c9b64fd3a8750b09e4932d6a: Neither blocks nor log segments found. Creating new log.
I20260926 20:55:04.108270 22176 log.cc:824] T 00000000000000000000000000000000 P 8375fe14c9b64fd3a8750b09e4932d6a: Log is configured to *not* fsync() on all Append() calls
I20260926 20:55:04.109093 22176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8375fe14c9b64fd3a8750b09e4932d6a: No bootstrap required, opened a new log
I20260926 20:55:04.108513 22150 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "8375fe14c9b64fd3a8750b09e4932d6a"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58290
I20260926 20:55:04.109875 22326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b99d668819104d0b96de636d6b88a991: Bootstrap starting.
I20260926 20:55:04.110467 22176 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8375fe14c9b64fd3a8750b09e4932d6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8375fe14c9b64fd3a8750b09e4932d6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46149 } } peers { permanent_uuid: "8dd574607f5d47debbb98db6c49e002a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40001 } } peers { permanent_uuid: "b99d668819104d0b96de636d6b88a991" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45851 } }
I20260926 20:55:04.110551 22326 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b99d668819104d0b96de636d6b88a991: Neither blocks nor log segments found. Creating new log.
I20260926 20:55:04.110605 22176 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8375fe14c9b64fd3a8750b09e4932d6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8375fe14c9b64fd3a8750b09e4932d6a, State: Initialized, Role: FOLLOWER
I20260926 20:55:04.110692 22176 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8375fe14c9b64fd3a8750b09e4932d6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8375fe14c9b64fd3a8750b09e4932d6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46149 } } peers { permanent_uuid: "8dd574607f5d47debbb98db6c49e002a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40001 } } peers { permanent_uuid: "b99d668819104d0b96de636d6b88a991" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45851 } }
I20260926 20:55:04.110828 22326 log.cc:824] T 00000000000000000000000000000000 P b99d668819104d0b96de636d6b88a991: Log is configured to *not* fsync() on all Append() calls
I20260926 20:55:04.110946 22333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8375fe14c9b64fd3a8750b09e4932d6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8375fe14c9b64fd3a8750b09e4932d6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46149 } } peers { permanent_uuid: "8dd574607f5d47debbb98db6c49e002a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40001 } } peers { permanent_uuid: "b99d668819104d0b96de636d6b88a991" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45851 } } }
I20260926 20:55:04.111008 22176 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8375fe14c9b64fd3a8750b09e4932d6a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:55:04.111047 22150 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8375fe14c9b64fd3a8750b09e4932d6a [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 20:55:04.111055 22333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8375fe14c9b64fd3a8750b09e4932d6a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:55:04.111121 22150 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8375fe14c9b64fd3a8750b09e4932d6a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:55:04.112010 22150 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8375fe14c9b64fd3a8750b09e4932d6a [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8375fe14c9b64fd3a8750b09e4932d6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46149 } } peers { permanent_uuid: "8dd574607f5d47debbb98db6c49e002a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40001 } } peers { permanent_uuid: "b99d668819104d0b96de636d6b88a991" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45851 } }
I20260926 20:55:04.112269 22150 leader_election.cc:288] T 00000000000000000000000000000000 P 8375fe14c9b64fd3a8750b09e4932d6a [CANDIDATE]: Term 1 election: Requested vote from peers 8dd574607f5d47debbb98db6c49e002a (127.0.98.61:40001), b99d668819104d0b96de636d6b88a991 (127.0.98.60:45851)
W20260926 20:55:04.113324 22345 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8375fe14c9b64fd3a8750b09e4932d6a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:55:04.113401 22345 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:55:04.114327 22326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b99d668819104d0b96de636d6b88a991: No bootstrap required, opened a new log
I20260926 20:55:04.113983 22108 master_runner.cc:186] Error getting master registration for 127.0.98.61:40001: OK, instance_id { permanent_uuid: "8dd574607f5d47debbb98db6c49e002a" instance_seqno: 1790456103904070 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 20:55:04.116447 22108 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 20:55:04.116443 22326 raft_consensus.cc:348] T 00000000000000000000000000000000 P b99d668819104d0b96de636d6b88a991 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8375fe14c9b64fd3a8750b09e4932d6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46149 } } peers { permanent_uuid: "8dd574607f5d47debbb98db6c49e002a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40001 } } peers { permanent_uuid: "b99d668819104d0b96de636d6b88a991" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45851 } }
I20260926 20:55:04.116598 22326 raft_consensus.cc:729] T 00000000000000000000000000000000 P b99d668819104d0b96de636d6b88a991 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b99d668819104d0b96de636d6b88a991, State: Initialized, Role: FOLLOWER
I20260926 20:55:04.116681 22326 consensus_queue.cc:260] T 00000000000000000000000000000000 P b99d668819104d0b96de636d6b88a991 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8375fe14c9b64fd3a8750b09e4932d6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46149 } } peers { permanent_uuid: "8dd574607f5d47debbb98db6c49e002a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40001 } } peers { permanent_uuid: "b99d668819104d0b96de636d6b88a991" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45851 } }
I20260926 20:55:04.116979 22326 sys_catalog.cc:564] T 00000000000000000000000000000000 P b99d668819104d0b96de636d6b88a991 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:55:04.117063 22300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8375fe14c9b64fd3a8750b09e4932d6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b99d668819104d0b96de636d6b88a991"
I20260926 20:55:04.117187 22300 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b99d668819104d0b96de636d6b88a991 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:55:04.118074 22300 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b99d668819104d0b96de636d6b88a991 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8375fe14c9b64fd3a8750b09e4932d6a in term 1.
I20260926 20:55:04.118287 22124 leader_election.cc:302] T 00000000000000000000000000000000 P 8375fe14c9b64fd3a8750b09e4932d6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8375fe14c9b64fd3a8750b09e4932d6a, b99d668819104d0b96de636d6b88a991; no voters:
I20260926 20:55:04.118409 22333 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8375fe14c9b64fd3a8750b09e4932d6a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:55:04.118563 22333 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8375fe14c9b64fd3a8750b09e4932d6a [term 1 LEADER]: Becoming Leader. State: Replica: 8375fe14c9b64fd3a8750b09e4932d6a, State: Running, Role: LEADER
I20260926 20:55:04.118644 22333 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8375fe14c9b64fd3a8750b09e4932d6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8375fe14c9b64fd3a8750b09e4932d6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46149 } } peers { permanent_uuid: "8dd574607f5d47debbb98db6c49e002a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40001 } } peers { permanent_uuid: "b99d668819104d0b96de636d6b88a991" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45851 } }
I20260926 20:55:04.119053 22333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8375fe14c9b64fd3a8750b09e4932d6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8375fe14c9b64fd3a8750b09e4932d6a. Latest consensus state: current_term: 1 leader_uuid: "8375fe14c9b64fd3a8750b09e4932d6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8375fe14c9b64fd3a8750b09e4932d6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46149 } } peers { permanent_uuid: "8dd574607f5d47debbb98db6c49e002a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40001 } } peers { permanent_uuid: "b99d668819104d0b96de636d6b88a991" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45851 } } }
I20260926 20:55:04.119134 22333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8375fe14c9b64fd3a8750b09e4932d6a [sys.catalog]: This master's current role is: LEADER
W20260926 20:55:04.119311 22365 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b99d668819104d0b96de636d6b88a991: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:55:04.119370 22365 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:55:04.119549 22354 sys_catalog.cc:455] T 00000000000000000000000000000000 P b99d668819104d0b96de636d6b88a991 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8375fe14c9b64fd3a8750b09e4932d6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46149 } } peers { permanent_uuid: "8dd574607f5d47debbb98db6c49e002a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40001 } } peers { permanent_uuid: "b99d668819104d0b96de636d6b88a991" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45851 } } }
I20260926 20:55:04.119693 22354 sys_catalog.cc:458] T 00000000000000000000000000000000 P b99d668819104d0b96de636d6b88a991 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:55:04.120272 22369 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:55:04.121408 22369 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:55:04.120873 22258 master_runner.cc:186] Error getting master registration for 127.0.98.61:40001: OK, instance_id { permanent_uuid: "8dd574607f5d47debbb98db6c49e002a" instance_seqno: 1790456103904070 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 20:55:04.123360 22258 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 20:55:04.125231 22255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:04.113625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42592 (local address 127.0.98.61:40001)
0926 20:55:04.113719 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:04.113721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:04.116670 (+ 2949us) server_negotiation.cc:407] Connection header received
0926 20:55:04.116766 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:04.116767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:04.116777 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:04.116826 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:04.117628 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:04.118336 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:04.119398 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:04.119705 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:04.119790 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:04.121836 (+ 2046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:04.121837 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:04.121838 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:04.121846 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:04.123918 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:04.124323 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:04.124324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:04.124324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:04.124390 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:04.124659 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:04.124660 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:04.124660 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:04.124795 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:04.124899 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:04.125055 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 20:55:04.125142 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 20:55:04.130643 22330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:04.113725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52688 (local address 127.0.98.60:45851)
0926 20:55:04.113766 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:04.113768 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:04.123146 (+ 9378us) server_negotiation.cc:407] Connection header received
0926 20:55:04.123376 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:04.123377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:04.123387 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:04.123418 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:04.124105 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:04.124810 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:04.125840 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:04.126090 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:04.126181 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:04.127609 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:04.127610 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:04.127610 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:04.127619 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:04.129408 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:04.129799 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:04.129800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:04.129800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:04.129858 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:04.130079 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:04.130080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:04.130080 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:04.130206 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:04.130276 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:04.130468 (+ 192us) server_negotiation.cc:299] Negotiation successful
0926 20:55:04.130546 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 20:55:04.135850 22179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:04.113677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58296 (local address 127.0.98.62:46149)
0926 20:55:04.113749 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:04.113751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:04.123243 (+ 9492us) server_negotiation.cc:407] Connection header received
0926 20:55:04.123297 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:04.123299 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:04.123312 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:04.123372 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:04.128374 (+ 5002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:04.129158 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:04.130776 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:04.131054 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:04.131098 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:04.132736 (+ 1638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:04.132737 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:04.132737 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:04.132762 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:04.134400 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:04.134870 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:04.134871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:04.134871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:04.134936 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:04.135169 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:04.135170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:04.135171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:04.135321 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:04.135409 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:04.135618 (+ 209us) server_negotiation.cc:299] Negotiation successful
0926 20:55:04.135696 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 20:55:04.137609 22300 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b99d668819104d0b96de636d6b88a991 [term 1 FOLLOWER]: Refusing update from remote peer 8375fe14c9b64fd3a8750b09e4932d6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:55:04.137964 22366 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8375fe14c9b64fd3a8750b09e4932d6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b99d668819104d0b96de636d6b88a991" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:55:04.139441 22354 sys_catalog.cc:455] T 00000000000000000000000000000000 P b99d668819104d0b96de636d6b88a991 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8375fe14c9b64fd3a8750b09e4932d6a. Latest consensus state: current_term: 1 leader_uuid: "8375fe14c9b64fd3a8750b09e4932d6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8375fe14c9b64fd3a8750b09e4932d6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46149 } } peers { permanent_uuid: "8dd574607f5d47debbb98db6c49e002a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40001 } } peers { permanent_uuid: "b99d668819104d0b96de636d6b88a991" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45851 } } }
I20260926 20:55:04.139722 22354 sys_catalog.cc:458] T 00000000000000000000000000000000 P b99d668819104d0b96de636d6b88a991 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:55:04.140712 22333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8375fe14c9b64fd3a8750b09e4932d6a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8375fe14c9b64fd3a8750b09e4932d6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8375fe14c9b64fd3a8750b09e4932d6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46149 } } peers { permanent_uuid: "8dd574607f5d47debbb98db6c49e002a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40001 } } peers { permanent_uuid: "b99d668819104d0b96de636d6b88a991" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45851 } } }
I20260926 20:55:04.140800 22333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8375fe14c9b64fd3a8750b09e4932d6a [sys.catalog]: This master's current role is: LEADER
I20260926 20:55:04.141198 22251 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45851 } attempt: 3
I20260926 20:55:04.141355 22369 catalog_manager.cc:1347] Generated new cluster ID: caa127e6f7fd4b7fa16f986212bc07ce
I20260926 20:55:04.141399 22369 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:55:04.142315 22354 sys_catalog.cc:455] T 00000000000000000000000000000000 P b99d668819104d0b96de636d6b88a991 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8375fe14c9b64fd3a8750b09e4932d6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8375fe14c9b64fd3a8750b09e4932d6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46149 } } peers { permanent_uuid: "8dd574607f5d47debbb98db6c49e002a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40001 } } peers { permanent_uuid: "b99d668819104d0b96de636d6b88a991" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45851 } } }
I20260926 20:55:04.142402 22354 sys_catalog.cc:458] T 00000000000000000000000000000000 P b99d668819104d0b96de636d6b88a991 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:55:04.142511 22370 mvcc.cc:205] Tried to move back new op lower bound from 7333708202544050176 to 7333708202470612992. Current Snapshot: MvccSnapshot[applied={T|T < 7333708202544050176 or (T in {7333708202544050176})}]
I20260926 20:55:04.142761 22371 mvcc.cc:205] Tried to move back new op lower bound from 7333708202544050176 to 7333708202470612992. Current Snapshot: MvccSnapshot[applied={T|T < 7333708202544050176}]
Sep 26 20:55:04 dist-test-slave-hwlj krb5kdc[22095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 20:55:04.150040 22330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:04.141677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52696 (local address 127.0.98.60:45851)
0926 20:55:04.141713 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:04.141716 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:04.141724 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:55:04.141747 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:04.141748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:04.141759 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:04.141822 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:04.142118 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:04.142858 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:04.143860 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:04.144170 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:04.144237 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:04.146494 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:04.146496 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:04.146496 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:04.146503 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:04.148803 (+ 2300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:04.149210 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:04.149211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:04.149211 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:04.149281 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:04.149491 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:04.149492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:04.149492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:04.149637 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:04.149701 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:04.149864 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 20:55:04.149948 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260926 20:55:04.150061 22253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:04.141473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45851 (local address 127.0.0.1:52696)
0926 20:55:04.141559 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:04.141570 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:04.141611 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:04.141827 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:04.141828 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:04.141997 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:55:04.142042 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:04.142043 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:04.142937 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:04.142939 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:04.143713 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:04.143715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:04.144247 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:04.144249 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:04.144309 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:04.144858 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:55:04.144866 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:55:04.146404 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:55:04.148883 (+ 2479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:04.148884 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:04.148885 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:04.149124 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:04.149342 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:04.149343 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:04.149343 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:04.149397 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:04.149708 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:55:04.149709 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:55:04.149775 (+ 66us) client_negotiation.cc:769] Sending connection context
0926 20:55:04.149837 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 20:55:04.149945 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 20:55:04.152132 22251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dd574607f5d47debbb98db6c49e002a: Bootstrap starting.
I20260926 20:55:04.152937 22251 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8dd574607f5d47debbb98db6c49e002a: Neither blocks nor log segments found. Creating new log.
I20260926 20:55:04.153249 22251 log.cc:824] T 00000000000000000000000000000000 P 8dd574607f5d47debbb98db6c49e002a: Log is configured to *not* fsync() on all Append() calls
I20260926 20:55:04.154690 22251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dd574607f5d47debbb98db6c49e002a: No bootstrap required, opened a new log
I20260926 20:55:04.157032 22251 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8dd574607f5d47debbb98db6c49e002a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8375fe14c9b64fd3a8750b09e4932d6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46149 } } peers { permanent_uuid: "8dd574607f5d47debbb98db6c49e002a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40001 } } peers { permanent_uuid: "b99d668819104d0b96de636d6b88a991" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45851 } }
I20260926 20:55:04.157195 22251 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8dd574607f5d47debbb98db6c49e002a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dd574607f5d47debbb98db6c49e002a, State: Initialized, Role: FOLLOWER
I20260926 20:55:04.157330 22251 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8dd574607f5d47debbb98db6c49e002a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8375fe14c9b64fd3a8750b09e4932d6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46149 } } peers { permanent_uuid: "8dd574607f5d47debbb98db6c49e002a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40001 } } peers { permanent_uuid: "b99d668819104d0b96de636d6b88a991" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45851 } }
I20260926 20:55:04.157692 22376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dd574607f5d47debbb98db6c49e002a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8375fe14c9b64fd3a8750b09e4932d6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46149 } } peers { permanent_uuid: "8dd574607f5d47debbb98db6c49e002a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40001 } } peers { permanent_uuid: "b99d668819104d0b96de636d6b88a991" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45851 } } }
I20260926 20:55:04.157788 22224 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8dd574607f5d47debbb98db6c49e002a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:55:04.157811 22225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8375fe14c9b64fd3a8750b09e4932d6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8dd574607f5d47debbb98db6c49e002a"
I20260926 20:55:04.157738 22251 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8dd574607f5d47debbb98db6c49e002a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:55:04.158015 22376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dd574607f5d47debbb98db6c49e002a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:55:04.158937 22224 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8dd574607f5d47debbb98db6c49e002a [term 1 FOLLOWER]: Refusing update from remote peer 8375fe14c9b64fd3a8750b09e4932d6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:55:04.161419 22333 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8375fe14c9b64fd3a8750b09e4932d6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dd574607f5d47debbb98db6c49e002a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:55:04.162385 22225 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8dd574607f5d47debbb98db6c49e002a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8375fe14c9b64fd3a8750b09e4932d6a in term 1.
I20260926 20:55:04.163798 22390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dd574607f5d47debbb98db6c49e002a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8375fe14c9b64fd3a8750b09e4932d6a. Latest consensus state: current_term: 1 leader_uuid: "8375fe14c9b64fd3a8750b09e4932d6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8375fe14c9b64fd3a8750b09e4932d6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46149 } } peers { permanent_uuid: "8dd574607f5d47debbb98db6c49e002a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40001 } } peers { permanent_uuid: "b99d668819104d0b96de636d6b88a991" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45851 } } }
I20260926 20:55:04.165057 22392 catalog_manager.cc:1259] Loaded cluster ID: caa127e6f7fd4b7fa16f986212bc07ce
I20260926 20:55:04.165100 22392 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8dd574607f5d47debbb98db6c49e002a: loading cluster ID for follower catalog manager: success
W20260926 20:55:04.165181 22392 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 8dd574607f5d47debbb98db6c49e002a: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260926 20:55:04.165216 22392 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:55:04.163971 22390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dd574607f5d47debbb98db6c49e002a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:55:04.164031 22376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dd574607f5d47debbb98db6c49e002a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8375fe14c9b64fd3a8750b09e4932d6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8375fe14c9b64fd3a8750b09e4932d6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46149 } } peers { permanent_uuid: "8dd574607f5d47debbb98db6c49e002a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40001 } } peers { permanent_uuid: "b99d668819104d0b96de636d6b88a991" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45851 } } }
I20260926 20:55:04.166956 22376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dd574607f5d47debbb98db6c49e002a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:55:04.169564 22366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8375fe14c9b64fd3a8750b09e4932d6a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8375fe14c9b64fd3a8750b09e4932d6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8375fe14c9b64fd3a8750b09e4932d6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46149 } } peers { permanent_uuid: "8dd574607f5d47debbb98db6c49e002a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40001 } } peers { permanent_uuid: "b99d668819104d0b96de636d6b88a991" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45851 } } }
I20260926 20:55:04.169680 22366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8375fe14c9b64fd3a8750b09e4932d6a [sys.catalog]: This master's current role is: LEADER
I20260926 20:55:04.171087 22369 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:55:04.171950 22369 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:55:04.184006 22369 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8375fe14c9b64fd3a8750b09e4932d6a: Generated new TSK 0
I20260926 20:55:04.184317 22369 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 20:55:04.209709 22179 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:55:04.207914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58300 (local address 127.0.98.62:46149)
0926 20:55:04.207969 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:04.207971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:04.208908 (+ 937us) server_negotiation.cc:407] Connection header received
0926 20:55:04.209023 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:04.209025 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:04.209037 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:04.209071 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:04.209625 (+ 554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58300: BlockingRecv error: recv got EOF from 127.0.0.1:58300 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260926 20:55:04.210839 22255 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:55:04.208393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42594 (local address 127.0.98.61:40001)
0926 20:55:04.208446 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:04.208449 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:04.210201 (+ 1752us) server_negotiation.cc:407] Connection header received
0926 20:55:04.210311 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:04.210313 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:04.210326 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:04.210358 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:04.210752 (+ 394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42594: BlockingRecv error: recv got EOF from 127.0.0.1:42594 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260926 20:55:04.211910 22330 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:55:04.208796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52710 (local address 127.0.98.60:45851)
0926 20:55:04.208848 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:04.208850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:04.211277 (+ 2427us) server_negotiation.cc:407] Connection header received
0926 20:55:04.211319 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:04.211321 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:04.211335 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:04.211410 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:04.211813 (+ 403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52710: BlockingRecv error: recv got EOF from 127.0.0.1:52710 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260926 20:55:04.214635 22255 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:55:04.213738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42598 (local address 127.0.98.61:40001)
0926 20:55:04.213783 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:04.213785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:04.214011 (+ 226us) server_negotiation.cc:407] Connection header received
0926 20:55:04.214130 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:04.214131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:04.214152 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:04.214183 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:04.214565 (+ 382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42598: BlockingRecv error: recv got EOF from 127.0.0.1:42598 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 20:55:04.218152 22255 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:55:04.217253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42612 (local address 127.0.98.61:40001)
0926 20:55:04.217300 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:04.217302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:04.217553 (+ 251us) server_negotiation.cc:407] Connection header received
0926 20:55:04.217660 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:04.217661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:04.217674 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:04.217703 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:04.218070 (+ 367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42612: BlockingRecv error: recv got EOF from 127.0.0.1:42612 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260926 20:55:04.222626 22255 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:55:04.221736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42618 (local address 127.0.98.61:40001)
0926 20:55:04.221787 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:04.221788 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:04.222034 (+ 246us) server_negotiation.cc:407] Connection header received
0926 20:55:04.222142 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:04.222143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:04.222154 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:04.222184 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:04.222568 (+ 384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42618: BlockingRecv error: recv got EOF from 127.0.0.1:42618 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 20:55:04.229095 22255 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:55:04.228168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42632 (local address 127.0.98.61:40001)
0926 20:55:04.228211 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:04.228214 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:04.228474 (+ 260us) server_negotiation.cc:407] Connection header received
0926 20:55:04.228599 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:04.228601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:04.228613 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:04.228643 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:04.229022 (+ 379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42632: BlockingRecv error: recv got EOF from 127.0.0.1:42632 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 20:55:04.239966 22255 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:55:04.238813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42638 (local address 127.0.98.61:40001)
0926 20:55:04.238872 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:04.238875 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:04.239154 (+ 279us) server_negotiation.cc:407] Connection header received
0926 20:55:04.239266 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:04.239267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:04.239298 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:04.239330 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:04.239903 (+ 573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42638: BlockingRecv error: recv got EOF from 127.0.0.1:42638 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 20:55:04.258677 22255 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:55:04.257659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42654 (local address 127.0.98.61:40001)
0926 20:55:04.257711 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:04.257713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:04.258012 (+ 299us) server_negotiation.cc:407] Connection header received
0926 20:55:04.258129 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:04.258130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:04.258143 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:04.258174 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:04.258611 (+ 437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42654: BlockingRecv error: recv got EOF from 127.0.0.1:42654 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260926 20:55:04.293239 22255 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:55:04.292369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42668 (local address 127.0.98.61:40001)
0926 20:55:04.292413 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:04.292415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:04.292651 (+ 236us) server_negotiation.cc:407] Connection header received
0926 20:55:04.292752 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:04.292753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:04.292764 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:04.292796 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:04.293175 (+ 379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42668: BlockingRecv error: recv got EOF from 127.0.0.1:42668 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 20:55:04.359661 22255 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:55:04.358908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42676 (local address 127.0.98.61:40001)
0926 20:55:04.358963 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:04.358965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:04.359073 (+ 108us) server_negotiation.cc:407] Connection header received
0926 20:55:04.359107 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:04.359108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:04.359135 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:04.359224 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:04.359583 (+ 359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42676: BlockingRecv error: recv got EOF from 127.0.0.1:42676 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260926 20:55:04.489682 22255 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:55:04.488833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42682 (local address 127.0.98.61:40001)
0926 20:55:04.488907 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:04.488909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:04.489024 (+ 115us) server_negotiation.cc:407] Connection header received
0926 20:55:04.489055 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:04.489055 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:04.489065 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:04.489129 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:04.489594 (+ 465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42682: BlockingRecv error: recv got EOF from 127.0.0.1:42682 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260926 20:55:04.747920 22255 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:55:04.747139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42696 (local address 127.0.98.61:40001)
0926 20:55:04.747257 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:04.747259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:04.747276 (+ 17us) server_negotiation.cc:407] Connection header received
0926 20:55:04.747305 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:04.747306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:04.747317 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:04.747399 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:04.747840 (+ 441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42696: BlockingRecv error: recv got EOF from 127.0.0.1:42696 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 20:55:05.119823 22365 catalog_manager.cc:1259] Loaded cluster ID: caa127e6f7fd4b7fa16f986212bc07ce
I20260926 20:55:05.119904 22365 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b99d668819104d0b96de636d6b88a991: loading cluster ID for follower catalog manager: success
I20260926 20:55:05.120695 22365 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b99d668819104d0b96de636d6b88a991: acquiring CA information for follower catalog manager: success
I20260926 20:55:05.120944 22365 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b99d668819104d0b96de636d6b88a991: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 20:55:05.166314 22392 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8dd574607f5d47debbb98db6c49e002a: acquiring CA information for follower catalog manager: success
I20260926 20:55:05.166652 22392 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8dd574607f5d47debbb98db6c49e002a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 20:55:05.263851 22458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:05.261029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42702 (local address 127.0.98.61:40001)
0926 20:55:05.261164 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:05.261166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:05.261242 (+ 76us) server_negotiation.cc:407] Connection header received
0926 20:55:05.261313 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:05.261314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:05.261372 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:05.261468 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:55:05.261714 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:05.262240 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:05.263017 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:05.263219 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:05.263276 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:05.263448 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:55:05.263563 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:55:05.263636 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:05.263683 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 20:55:05.263702 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":69,"threads_started":1}
I20260926 20:55:05.268363 22464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:05.265866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52716 (local address 127.0.98.60:45851)
0926 20:55:05.266005 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:05.266006 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:05.266015 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:55:05.266082 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:05.266082 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:05.266121 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:05.266192 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:55:05.266280 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:05.266796 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:05.267518 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:05.267724 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:05.267890 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:05.268024 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:55:05.268134 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:55:05.268191 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:05.268223 (+ 32us) server_negotiation.cc:299] Negotiation successful
0926 20:55:05.268240 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":83,"threads_started":1}
I20260926 20:55:05.273248 22471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:05.270143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58314 (local address 127.0.98.62:46149)
0926 20:55:05.270259 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:05.270261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:05.270391 (+ 130us) server_negotiation.cc:407] Connection header received
0926 20:55:05.270470 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:05.270471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:05.270525 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:05.270603 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:55:05.270779 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:05.271521 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:05.272368 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:05.272640 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:05.272703 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:05.272849 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:55:05.272968 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:55:05.273027 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:05.273124 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 20:55:05.273156 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":64,"threads_started":1}
I20260926 20:55:05.273376 22458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:05.270164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42704 (local address 127.0.98.61:40001)
0926 20:55:05.270206 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:05.270208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:05.270371 (+ 163us) server_negotiation.cc:407] Connection header received
0926 20:55:05.270487 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:05.270488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:05.270498 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:05.270560 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:55:05.270704 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:05.271455 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:05.272583 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:05.272869 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:05.272935 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:05.273043 (+ 108us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:55:05.273079 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:55:05.273128 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:05.273277 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 20:55:05.273303 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 20:55:05.274057 22464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:05.270415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52732 (local address 127.0.98.60:45851)
0926 20:55:05.270625 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:05.270627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:05.270634 (+ 7us) server_negotiation.cc:407] Connection header received
0926 20:55:05.270657 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:05.270658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:05.270668 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:05.270718 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:55:05.270886 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:05.271352 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:05.272973 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:05.273271 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:05.273331 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:05.273766 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:55:05.273822 (+ 56us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:55:05.273858 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:05.273961 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 20:55:05.273988 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178}
I20260926 20:55:05.275137 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 22108
I20260926 20:55:05.284093 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 22183
I20260926 20:55:05.289078 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 22258
2026-09-26T20:55:05Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2837 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:55:05 dist-test-slave-hwlj krb5kdc[22475](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:55:05 dist-test-slave-hwlj krb5kdc[22475](info): set up 2 sockets
Sep 26 20:55:05 dist-test-slave-hwlj krb5kdc[22475](info): commencing operation
krb5kdc: starting...
W20260926 20:55:06.332064 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/krb5kdc/test-user.keytab.
Sep 26 20:55:06 dist-test-slave-hwlj krb5kdc[22475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:55:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:55:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 20:55:06.455190 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39883
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39883,127.0.98.61:39101,127.0.98.60:43065
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:55:06.537099 22488 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 20:55:06.537333 22488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:55:06.537352 22488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:55:06.537374 22488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:55:06.538607 22488 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:55:06.538658 22488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:55:06.538673 22488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:55:06.538686 22488 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:55:06.538699 22488 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:55:06.540282 22488 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39883,127.0.98.61:39101,127.0.98.60:43065
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:55:06.540570 22488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:55:06.540832 22488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:55:06.545938 22496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:06.545923 22493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:06.545950 22494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:06.546386 22488 server_base.cc:1034] running on GCE node
I20260926 20:55:06.546710 22488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:55:06.546999 22488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:55:06.548180 22488 hybrid_clock.cc:648] HybridClock initialized: now 1790456106548166 us; error 52 us; skew 500 ppm
Sep 26 20:55:06 dist-test-slave-hwlj krb5kdc[22475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456106, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:55:06.551064 22488 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 20:55:06.551596 22488 webserver.cc:460] Webserver started at http://127.0.98.62:35263/ using document root <none> and password file <none>
I20260926 20:55:06.551913 22488 fs_manager.cc:359] Metadata directory not provided
I20260926 20:55:06.551985 22488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:55:06.552124 22488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:55:06.553241 22488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/minicluster-data/master-0/data/instance:
uuid: "87c8dfe59b704e0aafc579e2b3b9d066"
format_stamp: "Formatted at 2026-09-26 20:55:06 on dist-test-slave-hwlj"
server_key: "0c1fe07d1554d09e9bb290aec756af75"
server_key_iv: "43bd795b655f2b828ae96a842b10d622"
server_key_version: "encryptionkey@0"
I20260926 20:55:06.553628 22488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/minicluster-data/master-0/wal/instance:
uuid: "87c8dfe59b704e0aafc579e2b3b9d066"
format_stamp: "Formatted at 2026-09-26 20:55:06 on dist-test-slave-hwlj"
server_key: "0c1fe07d1554d09e9bb290aec756af75"
server_key_iv: "43bd795b655f2b828ae96a842b10d622"
server_key_version: "encryptionkey@0"
I20260926 20:55:06.555004 22488 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 20:55:06.555864 22503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:06.556036 22488 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:55:06.556151 22488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/minicluster-data/master-0/wal
uuid: "87c8dfe59b704e0aafc579e2b3b9d066"
format_stamp: "Formatted at 2026-09-26 20:55:06 on dist-test-slave-hwlj"
server_key: "0c1fe07d1554d09e9bb290aec756af75"
server_key_iv: "43bd795b655f2b828ae96a842b10d622"
server_key_version: "encryptionkey@0"
I20260926 20:55:06.556231 22488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:55:06.571476 22488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:55:06.573921 22488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:55:06.574098 22488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:55:06.578379 22488 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39883
I20260926 20:55:06.578413 22555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39883 every 8 connection(s)
I20260926 20:55:06.578785 22488 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/minicluster-data/master-0/data/info.pb
I20260926 20:55:06.579886 22556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:55:06.580215 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 22488
I20260926 20:55:06.580322 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/minicluster-data/master-0/wal/instance
I20260926 20:55:06.580556 392 external_mini_cluster.cc:1428] Setting key 2635ca573f7efab4b198ba84ed7c855f
I20260926 20:55:06.582345 22556 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39883 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 26 20:55:06 dist-test-slave-hwlj krb5kdc[22475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456106, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:55:06.596375 22559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:06.583144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39883 (local address 127.0.0.1:60650)
0926 20:55:06.583326 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:06.583370 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:06.583938 (+ 568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:06.584256 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:06.584259 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:06.584653 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:55:06.584842 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:06.584847 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:06.586534 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:06.586535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:06.587243 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:06.587245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:06.588007 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:06.588009 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:06.588071 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:06.588639 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:55:06.588649 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:55:06.592327 (+ 3678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:55:06.594624 (+ 2297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:06.594631 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:06.594632 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:06.594849 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:06.595155 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:06.595157 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:06.595157 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:06.595258 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:06.595705 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:55:06.595710 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:55:06.595825 (+ 115us) client_negotiation.cc:769] Sending connection context
0926 20:55:06.595933 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 20:55:06.596188 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":91,"thread_start_us":100,"threads_started":1}
I20260926 20:55:06.596378 22560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:06.583372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60650 (local address 127.0.98.62:39883)
0926 20:55:06.583523 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:06.583529 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:06.583568 (+ 39us) server_negotiation.cc:407] Connection header received
0926 20:55:06.584100 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:06.584103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:06.584127 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:06.584243 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:06.584963 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:06.586420 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:06.587334 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:06.587882 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:06.588016 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:06.592431 (+ 4415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:06.592445 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:06.592446 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:06.592454 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:06.594522 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:06.594941 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:06.594945 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:06.594945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:06.595060 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:06.595370 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:06.595371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:06.595372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:06.595620 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:06.595724 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:06.595971 (+ 247us) server_negotiation.cc:299] Negotiation successful
0926 20:55:06.596120 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":68,"threads_started":1}
I20260926 20:55:06.597282 22556 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39101 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:55:06.597659 22559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:06.597539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39101 (local address 127.0.0.1:40548)
0926 20:55:06.597589 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:06.597634 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 20:55:06.597832 22504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39101: connect: Connection refused (error 111)
I20260926 20:55:06.598301 22559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:06.598203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39101 (local address 127.0.0.1:40560)
0926 20:55:06.598251 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:06.598280 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 20:55:06.598453 22556 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39101: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39101: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260926 20:55:06.609283 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39101
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39883,127.0.98.61:39101,127.0.98.60:43065
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:55:06.647712 22556 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39101 } attempt: 1
I20260926 20:55:06.648424 22559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:06.648253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39101 (local address 127.0.0.1:40570)
0926 20:55:06.648331 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:06.648385 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 20:55:06.648833 22559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:06.648739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39101 (local address 127.0.0.1:40580)
0926 20:55:06.648784 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:06.648813 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 20:55:06.648947 22556 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39101: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39101: connect: Connection refused (error 111)
W20260926 20:55:06.691165 22563 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 20:55:06.691377 22563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:55:06.691394 22563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:55:06.691413 22563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:55:06.692605 22563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:55:06.692644 22563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:55:06.692658 22563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:55:06.692672 22563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:55:06.692683 22563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:55:06.694180 22563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39883,127.0.98.61:39101,127.0.98.60:43065
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:55:06.694430 22563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:55:06.694672 22563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:55:06.699376 22568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:06.699383 22571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:06.699390 22569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:06.699712 22563 server_base.cc:1034] running on GCE node
I20260926 20:55:06.699990 22563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:55:06.700270 22563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:55:06.701413 22563 hybrid_clock.cc:648] HybridClock initialized: now 1790456106701403 us; error 27 us; skew 500 ppm
Sep 26 20:55:06 dist-test-slave-hwlj krb5kdc[22475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456106, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:55:06.704010 22563 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260926 20:55:06.704509 22563 webserver.cc:460] Webserver started at http://127.0.98.61:43945/ using document root <none> and password file <none>
I20260926 20:55:06.704735 22563 fs_manager.cc:359] Metadata directory not provided
I20260926 20:55:06.704805 22563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:55:06.704959 22563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:55:06.706064 22563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/minicluster-data/master-1/data/instance:
uuid: "5ec66806433a47b3abd7f75f51703119"
format_stamp: "Formatted at 2026-09-26 20:55:06 on dist-test-slave-hwlj"
server_key: "e421b9dc5f7086d73e759ee7c141fded"
server_key_iv: "d15ca94cf54c091753fb69276379b780"
server_key_version: "encryptionkey@0"
I20260926 20:55:06.706470 22563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/minicluster-data/master-1/wal/instance:
uuid: "5ec66806433a47b3abd7f75f51703119"
format_stamp: "Formatted at 2026-09-26 20:55:06 on dist-test-slave-hwlj"
server_key: "e421b9dc5f7086d73e759ee7c141fded"
server_key_iv: "d15ca94cf54c091753fb69276379b780"
server_key_version: "encryptionkey@0"
I20260926 20:55:06.708247 22563 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260926 20:55:06.709069 22578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:06.709299 22563 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:55:06.709410 22563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/minicluster-data/master-1/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/minicluster-data/master-1/wal
uuid: "5ec66806433a47b3abd7f75f51703119"
format_stamp: "Formatted at 2026-09-26 20:55:06 on dist-test-slave-hwlj"
server_key: "e421b9dc5f7086d73e759ee7c141fded"
server_key_iv: "d15ca94cf54c091753fb69276379b780"
server_key_version: "encryptionkey@0"
I20260926 20:55:06.709491 22563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:55:06.717141 22556 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39101 } attempt: 2
I20260926 20:55:06.717617 22559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:06.717452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39101 (local address 127.0.0.1:40594)
0926 20:55:06.717521 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:06.717570 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 20:55:06.717998 22559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:06.717901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39101 (local address 127.0.0.1:40602)
0926 20:55:06.717948 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:06.717976 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 20:55:06.718115 22556 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39101: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39101: connect: Connection refused (error 111)
I20260926 20:55:06.725317 22563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:55:06.727731 22563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:55:06.727927 22563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:55:06.732133 22563 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:39101
I20260926 20:55:06.732183 22630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:39101 every 8 connection(s)
I20260926 20:55:06.732571 22563 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/minicluster-data/master-1/data/info.pb
I20260926 20:55:06.733700 22631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:55:06.734318 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 22563
I20260926 20:55:06.734411 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/minicluster-data/master-1/wal/instance
I20260926 20:55:06.734669 392 external_mini_cluster.cc:1428] Setting key ce0b93f6755aacfd145fb4cdeb6bd7c7
I20260926 20:55:06.735623 22631 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39883 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Sep 26 20:55:06 dist-test-slave-hwlj krb5kdc[22475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456106, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260926 20:55:06.745548 22560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:06.736533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60664 (local address 127.0.98.62:39883)
0926 20:55:06.736589 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:06.736591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:06.736830 (+ 239us) server_negotiation.cc:407] Connection header received
0926 20:55:06.737373 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:06.737375 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:06.737392 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:06.737443 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:06.738149 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:06.738634 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:06.739682 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:06.739951 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:06.740053 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:06.742259 (+ 2206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:06.742261 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:06.742261 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:06.742270 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:06.743938 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:06.744472 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:06.744473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:06.744473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:06.744541 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:06.744821 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:06.744822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:06.744822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:06.744963 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:06.745038 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:06.745352 (+ 314us) server_negotiation.cc:299] Negotiation successful
0926 20:55:06.745449 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 20:55:06.745774 22634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:06.736477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39883 (local address 127.0.0.1:60664)
0926 20:55:06.736698 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:06.736738 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:06.737251 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:06.737555 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:06.737558 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:06.737833 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:55:06.738034 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:06.738039 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:06.738735 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:06.738736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:06.739574 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:06.739576 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:06.740077 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:06.740079 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:06.740208 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:06.740758 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:55:06.740767 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:55:06.742159 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:55:06.744039 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:06.744048 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:06.744051 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:06.744357 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:06.744613 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:06.744614 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:06.744615 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:06.744743 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:06.745039 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:55:06.745048 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:55:06.745206 (+ 158us) client_negotiation.cc:769] Sending connection context
0926 20:55:06.745316 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 20:55:06.745507 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":114,"threads_started":1}
I20260926 20:55:06.746605 22631 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39101 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:55:06 dist-test-slave-hwlj krb5kdc[22475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456106, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 20:55:06.754623 22634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:06.746923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39101 (local address 127.0.0.1:40604)
0926 20:55:06.747000 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:06.747010 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:06.747051 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:06.747284 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:06.747285 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:06.747446 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:55:06.747496 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:06.747498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:06.748459 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:06.748460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:06.748980 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:06.748982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:06.749487 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:06.749488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:06.749558 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:06.750030 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:55:06.750036 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:55:06.751483 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:55:06.753459 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:06.753460 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:06.753461 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:06.753646 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:06.753875 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:06.753876 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:06.753876 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:06.753931 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:06.754263 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:55:06.754264 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:55:06.754306 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 20:55:06.754396 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 20:55:06.754505 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 20:55:06.754694 22636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:06.747001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40604 (local address 127.0.98.61:39101)
0926 20:55:06.747154 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:06.747159 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:06.747174 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:55:06.747191 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:06.747193 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:06.747203 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:06.747284 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:06.747576 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:06.748357 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:06.749080 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:06.749402 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:06.749492 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:06.751599 (+ 2107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:06.751607 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:06.751608 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:06.751615 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:06.753357 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:06.753741 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:06.753744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:06.753745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:06.753804 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:06.754000 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:06.754001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:06.754002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:06.754190 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:06.754273 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:06.754433 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 20:55:06.754559 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":131,"thread_start_us":102,"threads_started":1}
I20260926 20:55:06.755411 22631 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43065 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:55:06.755823 22634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:06.755665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43065 (local address 127.0.0.1:45766)
0926 20:55:06.755718 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:06.755798 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 20:55:06.755985 22580 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43065: connect: Connection refused (error 111)
I20260926 20:55:06.756439 22634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:06.756291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43065 (local address 127.0.0.1:45774)
0926 20:55:06.756364 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:06.756409 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260926 20:55:06.756544 22631 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43065: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43065: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260926 20:55:06.763041 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7paSjL/build/release/bin/kudu
/tmp/dist-test-task7paSjL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43065
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39883,127.0.98.61:39101,127.0.98.60:43065
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:55:06.805804 22631 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43065 } attempt: 1
I20260926 20:55:06.806356 22634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:06.806217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43065 (local address 127.0.0.1:45788)
0926 20:55:06.806282 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:06.806325 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 20:55:06.806771 22634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:06.806670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43065 (local address 127.0.0.1:45794)
0926 20:55:06.806715 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:06.806749 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 20:55:06.806914 22631 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43065: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43065: connect: Connection refused (error 111)
I20260926 20:55:06.809281 22556 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39101 } attempt: 3
Sep 26 20:55:06 dist-test-slave-hwlj krb5kdc[22475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456106, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 20:55:06.817416 22559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:06.809655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39101 (local address 127.0.0.1:40608)
0926 20:55:06.809703 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:06.809717 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:06.809797 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:06.810023 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:06.810024 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:06.810185 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:55:06.810253 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:06.810254 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:06.811267 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:06.811268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:06.811949 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:06.811951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:06.812479 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:06.812480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:06.812545 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:06.813073 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:55:06.813079 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:55:06.814323 (+ 1244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:55:06.816248 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:06.816249 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:06.816250 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:06.816470 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:06.816710 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:06.816711 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:06.816712 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:06.816779 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:06.817110 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:55:06.817112 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:55:06.817166 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 20:55:06.817253 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 20:55:06.817338 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260926 20:55:06.817480 22636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:06.809769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40608 (local address 127.0.98.61:39101)
0926 20:55:06.809852 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:06.809854 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:06.809865 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:55:06.809893 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:06.809894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:06.809905 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:06.809979 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:06.810342 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:06.811164 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:06.812071 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:06.812394 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:06.812465 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:06.814404 (+ 1939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:06.814406 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:06.814406 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:06.814414 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:06.816138 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:06.816550 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:06.816551 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:06.816551 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:06.816612 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:06.816851 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:06.816851 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:06.816852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:06.817016 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:06.817082 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:06.817297 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 20:55:06.817380 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 20:55:06.817909 22556 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43065 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:55:06.818277 22559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:06.818156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43065 (local address 127.0.0.1:45804)
0926 20:55:06.818212 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:06.818246 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 20:55:06.819254 22559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:06.819081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43065 (local address 127.0.0.1:45808)
0926 20:55:06.819201 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:06.819232 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260926 20:55:06.819391 22556 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43065: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43065: connect: Connection refused (error 111)
W20260926 20:55:06.848021 22638 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 20:55:06.848229 22638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:55:06.848248 22638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:55:06.848268 22638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:55:06.849336 22638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:55:06.849370 22638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:55:06.849385 22638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:55:06.849397 22638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:55:06.849409 22638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:55:06.850569 22556 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43065 } attempt: 1
I20260926 20:55:06.850997 22638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39883,127.0.98.61:39101,127.0.98.60:43065
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 19:57:09 UTC on ccycloud.quasar-whzewq.root.comops.site
build id 4675383
I20260926 20:55:06.851058 22559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:06.850904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43065 (local address 127.0.0.1:45824)
0926 20:55:06.850990 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:06.851024 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 20:55:06.851284 22638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:55:06.851568 22638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:55:06.851598 22559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:06.851450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43065 (local address 127.0.0.1:45828)
0926 20:55:06.851526 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:06.851566 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 20:55:06.851773 22556 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43065: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43065: connect: Connection refused (error 111)
W20260926 20:55:06.856812 22646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:06.856828 22638 server_base.cc:1034] running on GCE node
W20260926 20:55:06.856812 22643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:06.856804 22644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:06.857331 22638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:55:06.857604 22638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:55:06.858821 22638 hybrid_clock.cc:648] HybridClock initialized: now 1790456106858801 us; error 28 us; skew 500 ppm
Sep 26 20:55:06 dist-test-slave-hwlj krb5kdc[22475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456106, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:55:06.861446 22638 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260926 20:55:06.861827 22638 webserver.cc:460] Webserver started at http://127.0.98.60:45389/ using document root <none> and password file <none>
I20260926 20:55:06.862047 22638 fs_manager.cc:359] Metadata directory not provided
I20260926 20:55:06.862090 22638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:55:06.862180 22638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:55:06.863185 22638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/minicluster-data/master-2/data/instance:
uuid: "c4fbd2d28065430494041ef76250898e"
format_stamp: "Formatted at 2026-09-26 20:55:06 on dist-test-slave-hwlj"
server_key: "b9c59fc9e3b9cd17380743497eab7ee7"
server_key_iv: "2069efc838a5518fb0cd55c94b7f65a0"
server_key_version: "encryptionkey@0"
I20260926 20:55:06.863490 22638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/minicluster-data/master-2/wal/instance:
uuid: "c4fbd2d28065430494041ef76250898e"
format_stamp: "Formatted at 2026-09-26 20:55:06 on dist-test-slave-hwlj"
server_key: "b9c59fc9e3b9cd17380743497eab7ee7"
server_key_iv: "2069efc838a5518fb0cd55c94b7f65a0"
server_key_version: "encryptionkey@0"
I20260926 20:55:06.864904 22638 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 20:55:06.865674 22653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:06.865849 22638 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:55:06.865935 22638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/minicluster-data/master-2/data,/tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/minicluster-data/master-2/wal
uuid: "c4fbd2d28065430494041ef76250898e"
format_stamp: "Formatted at 2026-09-26 20:55:06 on dist-test-slave-hwlj"
server_key: "b9c59fc9e3b9cd17380743497eab7ee7"
server_key_iv: "2069efc838a5518fb0cd55c94b7f65a0"
server_key_version: "encryptionkey@0"
I20260926 20:55:06.865993 22638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:55:06.875057 22631 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43065 } attempt: 2
I20260926 20:55:06.875530 22634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:06.875388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43065 (local address 127.0.0.1:45830)
0926 20:55:06.875450 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:06.875489 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 20:55:06.875957 22634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:06.875844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43065 (local address 127.0.0.1:45846)
0926 20:55:06.875906 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:06.875937 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 20:55:06.876063 22631 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43065: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43065: connect: Connection refused (error 111)
I20260926 20:55:06.879181 22638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:55:06.881134 22638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:55:06.881292 22638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:55:06.885167 22638 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:43065
I20260926 20:55:06.885202 22705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:43065 every 8 connection(s)
I20260926 20:55:06.885569 22638 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/minicluster-data/master-2/data/info.pb
I20260926 20:55:06.886571 22706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:55:06.888319 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7paSjL/build/release/bin/kudu as pid 22638
I20260926 20:55:06.888230 22706 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39883 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:55:06.888442 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7paSjL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455725523895-392-0/minicluster-data/master-2/wal/instance
I20260926 20:55:06.888667 392 external_mini_cluster.cc:1428] Setting key 93efb5e3c993e73d122d6963548154cd
Sep 26 20:55:06 dist-test-slave-hwlj krb5kdc[22475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456106, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 26 20:55:06 dist-test-slave-hwlj krb5kdc[22475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 20:55:06.898303 22560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:06.888993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60680 (local address 127.0.98.62:39883)
0926 20:55:06.889193 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:06.889196 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:06.889278 (+ 82us) server_negotiation.cc:407] Connection header received
0926 20:55:06.889798 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:06.889799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:06.889811 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:06.889868 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:06.890614 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:06.891376 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:06.892605 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:06.892985 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:06.893058 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:06.895366 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:06.895368 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:06.895368 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:06.895377 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:06.896825 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:06.897277 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:06.897278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:06.897278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:06.897336 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:06.897585 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:06.897586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:06.897586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:06.897727 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:06.897795 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:06.898103 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 20:55:06.898184 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168}
I20260926 20:55:06.898452 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:06.888955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39883 (local address 127.0.0.1:60680)
0926 20:55:06.889129 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:06.889176 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:06.889709 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:06.889900 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:06.889902 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:06.890268 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:55:06.890505 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:06.890512 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:06.891457 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:06.891457 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:06.892520 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:06.892522 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:06.893094 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:06.893095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:06.893191 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:06.893719 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:55:06.893747 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:55:06.895284 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:55:06.896924 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:06.896930 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:06.896932 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:06.897170 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:06.897403 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:06.897404 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:06.897404 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:06.897508 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:06.897810 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:55:06.897816 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:55:06.897927 (+ 111us) client_negotiation.cc:769] Sending connection context
0926 20:55:06.898060 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 20:55:06.898210 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":84,"threads_started":1}
I20260926 20:55:06.898952 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:06.889045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45856 (local address 127.0.98.60:43065)
0926 20:55:06.889198 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:06.889203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:06.889300 (+ 97us) server_negotiation.cc:407] Connection header received
0926 20:55:06.889726 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:06.889737 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:06.889781 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:06.889880 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:06.890622 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:06.891707 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:06.892732 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:06.893181 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:06.893232 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:06.896150 (+ 2918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:06.896157 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:06.896157 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:06.896164 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:06.897675 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:06.897993 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:06.897996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:06.897996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:06.898056 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:06.898268 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:06.898269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:06.898269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:06.898416 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:06.898567 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:06.898771 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 20:55:06.898855 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":94,"threads_started":1}
I20260926 20:55:06.898991 22706 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39101 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:55:06 dist-test-slave-hwlj krb5kdc[22475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456106, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 26 20:55:06 dist-test-slave-hwlj krb5kdc[22475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 20:55:06.906003 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:06.899274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39101 (local address 127.0.0.1:40624)
0926 20:55:06.899327 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:06.899339 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:06.899379 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:06.899548 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:06.899549 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:06.899699 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:55:06.899734 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:06.899736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:06.900376 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:06.900377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:06.900919 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:06.900921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:06.901310 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:06.901311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:06.901371 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:06.901709 (+ 338us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:55:06.901714 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:55:06.903232 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:55:06.904968 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:06.904969 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:06.904970 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:06.905149 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:06.905355 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:06.905356 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:06.905356 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:06.905414 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:06.905684 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:55:06.905685 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:55:06.905729 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 20:55:06.905812 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 20:55:06.905906 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 20:55:06.906001 22636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:06.899379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40624 (local address 127.0.98.61:39101)
0926 20:55:06.899426 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:06.899428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:06.899435 (+ 7us) server_negotiation.cc:407] Connection header received
0926 20:55:06.899458 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:06.899459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:06.899468 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:06.899545 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:06.899820 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:06.900299 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:06.901001 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:06.901245 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:06.901292 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:06.903308 (+ 2016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:06.903309 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:06.903310 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:06.903316 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:06.904873 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:06.905224 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:06.905225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:06.905226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:06.905280 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:06.905483 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:06.905484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:06.905484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:06.905604 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:06.905681 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:06.905839 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 20:55:06.905910 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 20:55:06.906772 22706 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43065 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:55:06.907038 22560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:06.899873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60682 (local address 127.0.98.62:39883)
0926 20:55:06.899910 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:06.899912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:06.899920 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:55:06.899976 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:06.899977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:06.899987 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:06.900040 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:06.900520 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:06.900944 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:06.901791 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:06.901963 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:06.902014 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:06.904321 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:06.904323 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:06.904323 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:06.904330 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:06.905749 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:06.906149 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:06.906150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:06.906151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:06.906198 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:06.906400 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:06.906401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:06.906402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:06.906514 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:06.906692 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:06.906773 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 20:55:06.906849 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Sep 26 20:55:06 dist-test-slave-hwlj krb5kdc[22475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456106, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 26 20:55:06 dist-test-slave-hwlj krb5kdc[22475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 20:55:06.913951 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:06.907099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45866 (local address 127.0.98.60:43065)
0926 20:55:06.907136 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:06.907138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:06.907301 (+ 163us) server_negotiation.cc:407] Connection header received
0926 20:55:06.907372 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:06.907372 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:06.907383 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:06.907430 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:06.907690 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:06.908331 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:06.908964 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:06.909185 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:06.909274 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:06.911273 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:06.911274 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:06.911274 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:06.911279 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:06.912909 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:06.913277 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:06.913278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:06.913278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:06.913337 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:06.913540 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:06.913541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:06.913541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:06.913634 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:06.913681 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:06.913834 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 20:55:06.913895 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 20:55:06.914076 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:06.907038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43065 (local address 127.0.0.1:45866)
0926 20:55:06.907244 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:06.907253 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:06.907315 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:06.907430 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:06.907431 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:06.907588 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:55:06.907635 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:06.907636 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:06.908417 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:06.908418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:06.908886 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:06.908888 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:06.909273 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:06.909274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:06.909321 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:06.909738 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:55:06.909742 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:55:06.911197 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:55:06.912983 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:06.912984 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:06.912985 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:06.913198 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:06.913400 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:06.913401 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:06.913402 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:06.913466 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:06.913695 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:55:06.913696 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:55:06.913739 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 20:55:06.913819 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 20:55:06.913960 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":49}
I20260926 20:55:06.915112 22636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:06.907907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40638 (local address 127.0.98.61:39101)
0926 20:55:06.907946 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:06.907948 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:06.907955 (+ 7us) server_negotiation.cc:407] Connection header received
0926 20:55:06.907974 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:06.907975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:06.907982 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:06.908042 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:06.908454 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:06.908910 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:06.909568 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:06.909850 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:06.909902 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:06.912348 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:06.912350 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:06.912350 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:06.912357 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:06.914087 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:06.914432 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:06.914434 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:06.914434 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:06.914496 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:06.914705 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:06.914705 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:06.914705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:06.914798 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:06.914904 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:06.915002 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 20:55:06.915055 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 20:55:06.915900 22706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4fbd2d28065430494041ef76250898e: Bootstrap starting.
I20260926 20:55:06.916579 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:55:06.916913 22706 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4fbd2d28065430494041ef76250898e: Neither blocks nor log segments found. Creating new log.
I20260926 20:55:06.917197 22706 log.cc:824] T 00000000000000000000000000000000 P c4fbd2d28065430494041ef76250898e: Log is configured to *not* fsync() on all Append() calls
I20260926 20:55:06.918033 22706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4fbd2d28065430494041ef76250898e: No bootstrap required, opened a new log
I20260926 20:55:06.917445 22530 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "87c8dfe59b704e0aafc579e2b3b9d066"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60682
I20260926 20:55:06.920097 22706 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4fbd2d28065430494041ef76250898e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c8dfe59b704e0aafc579e2b3b9d066" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39883 } } peers { permanent_uuid: "5ec66806433a47b3abd7f75f51703119" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39101 } } peers { permanent_uuid: "c4fbd2d28065430494041ef76250898e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43065 } }
I20260926 20:55:06.920253 22706 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4fbd2d28065430494041ef76250898e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4fbd2d28065430494041ef76250898e, State: Initialized, Role: FOLLOWER
I20260926 20:55:06.920328 22706 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4fbd2d28065430494041ef76250898e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c8dfe59b704e0aafc579e2b3b9d066" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39883 } } peers { permanent_uuid: "5ec66806433a47b3abd7f75f51703119" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39101 } } peers { permanent_uuid: "c4fbd2d28065430494041ef76250898e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43065 } }
I20260926 20:55:06.920636 22706 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4fbd2d28065430494041ef76250898e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:55:06.920601 22712 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4fbd2d28065430494041ef76250898e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c8dfe59b704e0aafc579e2b3b9d066" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39883 } } peers { permanent_uuid: "5ec66806433a47b3abd7f75f51703119" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39101 } } peers { permanent_uuid: "c4fbd2d28065430494041ef76250898e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43065 } } }
I20260926 20:55:06.920712 22712 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4fbd2d28065430494041ef76250898e [sys.catalog]: This master's current role is: FOLLOWER
W20260926 20:55:06.922472 22723 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c4fbd2d28065430494041ef76250898e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:55:06.922525 22723 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:55:06.922725 22638 master_runner.cc:186] Error getting master registration for 127.0.98.62:39883: OK, instance_id { permanent_uuid: "87c8dfe59b704e0aafc579e2b3b9d066" instance_seqno: 1790456106577195 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 20:55:06.925542 22638 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 20:55:06.926950 22556 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43065 } attempt: 2
Sep 26 20:55:06 dist-test-slave-hwlj krb5kdc[22475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456106, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 20:55:06.935015 22559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:06.927230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43065 (local address 127.0.0.1:45874)
0926 20:55:06.927286 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:06.927297 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:06.927336 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:06.927605 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:06.927606 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:06.927806 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:55:06.927854 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:06.927855 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:06.928741 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:06.928742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:06.929518 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:06.929520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:06.929978 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:06.929979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:06.930035 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:06.930510 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:55:06.930515 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:55:06.932063 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:55:06.933877 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:06.933878 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:06.933879 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:06.934105 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:06.934326 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:06.934327 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:06.934328 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:06.934395 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:06.934697 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:55:06.934698 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:55:06.934745 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 20:55:06.934816 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 20:55:06.934910 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 20:55:06.935019 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:06.927426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45874 (local address 127.0.98.60:43065)
0926 20:55:06.927487 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:06.927489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:06.927497 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:55:06.927517 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:06.927518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:06.927528 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:06.927593 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:06.927927 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:06.928657 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:06.929602 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:06.929894 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:06.929967 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:06.932161 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:06.932162 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:06.932163 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:06.932169 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:06.933790 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:06.934190 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:06.934192 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:06.934192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:06.934257 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:06.934460 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:06.934461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:06.934462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:06.934626 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:06.934692 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:06.934851 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 20:55:06.934926 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 20:55:06.936509 22556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87c8dfe59b704e0aafc579e2b3b9d066: Bootstrap starting.
I20260926 20:55:06.937142 22556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87c8dfe59b704e0aafc579e2b3b9d066: Neither blocks nor log segments found. Creating new log.
I20260926 20:55:06.937424 22556 log.cc:824] T 00000000000000000000000000000000 P 87c8dfe59b704e0aafc579e2b3b9d066: Log is configured to *not* fsync() on all Append() calls
I20260926 20:55:06.938154 22556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87c8dfe59b704e0aafc579e2b3b9d066: No bootstrap required, opened a new log
I20260926 20:55:06.939172 22556 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87c8dfe59b704e0aafc579e2b3b9d066 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c8dfe59b704e0aafc579e2b3b9d066" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39883 } } peers { permanent_uuid: "5ec66806433a47b3abd7f75f51703119" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39101 } } peers { permanent_uuid: "c4fbd2d28065430494041ef76250898e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43065 } }
I20260926 20:55:06.939307 22556 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87c8dfe59b704e0aafc579e2b3b9d066 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87c8dfe59b704e0aafc579e2b3b9d066, State: Initialized, Role: FOLLOWER
I20260926 20:55:06.939400 22556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87c8dfe59b704e0aafc579e2b3b9d066 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c8dfe59b704e0aafc579e2b3b9d066" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39883 } } peers { permanent_uuid: "5ec66806433a47b3abd7f75f51703119" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39101 } } peers { permanent_uuid: "c4fbd2d28065430494041ef76250898e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43065 } }
I20260926 20:55:06.939715 22726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87c8dfe59b704e0aafc579e2b3b9d066 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c8dfe59b704e0aafc579e2b3b9d066" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39883 } } peers { permanent_uuid: "5ec66806433a47b3abd7f75f51703119" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39101 } } peers { permanent_uuid: "c4fbd2d28065430494041ef76250898e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43065 } } }
I20260926 20:55:06.939800 22530 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87c8dfe59b704e0aafc579e2b3b9d066 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 20:55:06.939785 22556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87c8dfe59b704e0aafc579e2b3b9d066 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:55:06.939851 22726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87c8dfe59b704e0aafc579e2b3b9d066 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:55:06.939886 22530 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87c8dfe59b704e0aafc579e2b3b9d066 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:55:06.940773 22530 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87c8dfe59b704e0aafc579e2b3b9d066 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c8dfe59b704e0aafc579e2b3b9d066" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39883 } } peers { permanent_uuid: "5ec66806433a47b3abd7f75f51703119" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39101 } } peers { permanent_uuid: "c4fbd2d28065430494041ef76250898e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43065 } }
I20260926 20:55:06.941048 22530 leader_election.cc:288] T 00000000000000000000000000000000 P 87c8dfe59b704e0aafc579e2b3b9d066 [CANDIDATE]: Term 1 election: Requested vote from peers 5ec66806433a47b3abd7f75f51703119 (127.0.98.61:39101), c4fbd2d28065430494041ef76250898e (127.0.98.60:43065)
I20260926 20:55:06.941342 22680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "87c8dfe59b704e0aafc579e2b3b9d066" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c4fbd2d28065430494041ef76250898e"
I20260926 20:55:06.941495 22680 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4fbd2d28065430494041ef76250898e [term 0 FOLLOWER]: Advancing to term 1
W20260926 20:55:06.942492 22739 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 87c8dfe59b704e0aafc579e2b3b9d066: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260926 20:55:06.942492 22680 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c4fbd2d28065430494041ef76250898e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87c8dfe59b704e0aafc579e2b3b9d066 in term 1.
W20260926 20:55:06.942533 22739 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:55:06.943198 22505 leader_election.cc:302] T 00000000000000000000000000000000 P 87c8dfe59b704e0aafc579e2b3b9d066 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87c8dfe59b704e0aafc579e2b3b9d066, c4fbd2d28065430494041ef76250898e; no voters:
I20260926 20:55:06.943207 22488 master_runner.cc:186] Error getting master registration for 127.0.98.61:39101: OK, instance_id { permanent_uuid: "5ec66806433a47b3abd7f75f51703119" instance_seqno: 1790456106730829 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 20:55:06.945614 22488 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 20:55:06.945681 22726 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87c8dfe59b704e0aafc579e2b3b9d066 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:55:06.945820 22726 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87c8dfe59b704e0aafc579e2b3b9d066 [term 1 LEADER]: Becoming Leader. State: Replica: 87c8dfe59b704e0aafc579e2b3b9d066, State: Running, Role: LEADER
I20260926 20:55:06.945905 22726 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87c8dfe59b704e0aafc579e2b3b9d066 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c8dfe59b704e0aafc579e2b3b9d066" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39883 } } peers { permanent_uuid: "5ec66806433a47b3abd7f75f51703119" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39101 } } peers { permanent_uuid: "c4fbd2d28065430494041ef76250898e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43065 } }
I20260926 20:55:06.946332 22726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87c8dfe59b704e0aafc579e2b3b9d066 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87c8dfe59b704e0aafc579e2b3b9d066. Latest consensus state: current_term: 1 leader_uuid: "87c8dfe59b704e0aafc579e2b3b9d066" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c8dfe59b704e0aafc579e2b3b9d066" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39883 } } peers { permanent_uuid: "5ec66806433a47b3abd7f75f51703119" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39101 } } peers { permanent_uuid: "c4fbd2d28065430494041ef76250898e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43065 } } }
I20260926 20:55:06.946422 22726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87c8dfe59b704e0aafc579e2b3b9d066 [sys.catalog]: This master's current role is: LEADER
I20260926 20:55:06.946959 22748 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:55:06.947191 22748 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:55:06.956429 22636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:06.946625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40640 (local address 127.0.98.61:39101)
0926 20:55:06.946695 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:06.946697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:06.948157 (+ 1460us) server_negotiation.cc:407] Connection header received
0926 20:55:06.948271 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:06.948273 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:06.948285 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:06.948316 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:06.949126 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:06.949947 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:06.951085 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:06.951387 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:06.951434 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:06.953304 (+ 1870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:06.953306 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:06.953306 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:06.953313 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:06.955027 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:06.955430 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:06.955431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:06.955431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:06.955504 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:06.955710 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:06.955711 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:06.955711 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:06.955881 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:06.955977 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:06.956223 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 20:55:06.956302 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 20:55:06.957512 22680 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c4fbd2d28065430494041ef76250898e [term 1 FOLLOWER]: Refusing update from remote peer 87c8dfe59b704e0aafc579e2b3b9d066: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:55:06.958681 22746 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 87c8dfe59b704e0aafc579e2b3b9d066 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4fbd2d28065430494041ef76250898e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:55:06.960372 22712 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4fbd2d28065430494041ef76250898e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87c8dfe59b704e0aafc579e2b3b9d066. Latest consensus state: current_term: 1 leader_uuid: "87c8dfe59b704e0aafc579e2b3b9d066" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c8dfe59b704e0aafc579e2b3b9d066" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39883 } } peers { permanent_uuid: "5ec66806433a47b3abd7f75f51703119" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39101 } } peers { permanent_uuid: "c4fbd2d28065430494041ef76250898e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43065 } } }
I20260926 20:55:06.960480 22712 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4fbd2d28065430494041ef76250898e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:55:06.967190 22631 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43065 } attempt: 3
I20260926 20:55:06.968087 22560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:06.944298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60698 (local address 127.0.98.62:39883)
0926 20:55:06.944343 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:06.944345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:06.951883 (+ 7538us) server_negotiation.cc:407] Connection header received
0926 20:55:06.951992 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:06.951992 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:06.952004 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:06.952052 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:06.956414 (+ 4362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:06.957235 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:06.958648 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:06.959015 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:06.959083 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:06.963352 (+ 4269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:06.963353 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:06.963354 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:06.963363 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:06.965521 (+ 2158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:06.966694 (+ 1173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:06.966696 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:06.966696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:06.966761 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:06.967144 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:06.967161 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:06.967161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:06.967307 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:06.967410 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:06.967859 (+ 449us) server_negotiation.cc:299] Negotiation successful
0926 20:55:06.967980 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 20:55:06.968603 22746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87c8dfe59b704e0aafc579e2b3b9d066 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "87c8dfe59b704e0aafc579e2b3b9d066" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c8dfe59b704e0aafc579e2b3b9d066" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39883 } } peers { permanent_uuid: "5ec66806433a47b3abd7f75f51703119" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39101 } } peers { permanent_uuid: "c4fbd2d28065430494041ef76250898e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43065 } } }
I20260926 20:55:06.968715 22746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87c8dfe59b704e0aafc579e2b3b9d066 [sys.catalog]: This master's current role is: LEADER
I20260926 20:55:06.969283 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:06.946626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45880 (local address 127.0.98.60:43065)
0926 20:55:06.946667 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:06.946670 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:06.957428 (+ 10758us) server_negotiation.cc:407] Connection header received
0926 20:55:06.957533 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:06.957534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:06.957546 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:06.957576 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:06.958317 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:06.958992 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:06.960081 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:06.960377 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:06.960422 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:06.965778 (+ 5356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:06.965780 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:06.965780 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:06.965787 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:06.967483 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:06.967994 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:06.967995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:06.967995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:06.968055 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:06.968521 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:06.968523 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:06.968523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:06.968658 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:06.968756 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:06.969128 (+ 372us) server_negotiation.cc:299] Negotiation successful
0926 20:55:06.969202 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 20:55:06.969395 22712 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4fbd2d28065430494041ef76250898e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "87c8dfe59b704e0aafc579e2b3b9d066" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c8dfe59b704e0aafc579e2b3b9d066" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39883 } } peers { permanent_uuid: "5ec66806433a47b3abd7f75f51703119" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39101 } } peers { permanent_uuid: "c4fbd2d28065430494041ef76250898e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43065 } } }
I20260926 20:55:06.969475 22712 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4fbd2d28065430494041ef76250898e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:55:06.969749 22749 mvcc.cc:205] Tried to move back new op lower bound from 7333708214094712832 to 7333708214051213312. Current Snapshot: MvccSnapshot[applied={T|T < 7333708214094712832 or (T in {7333708214094712832})}]
I20260926 20:55:06.969954 22750 mvcc.cc:205] Tried to move back new op lower bound from 7333708214094712832 to 7333708214051213312. Current Snapshot: MvccSnapshot[applied={T|T < 7333708214094712832}]
I20260926 20:55:06.970203 22748 catalog_manager.cc:1347] Generated new cluster ID: 2f9130daa378488cada713b82b6aa6e6
I20260926 20:55:06.970247 22748 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 20:55:06 dist-test-slave-hwlj krb5kdc[22475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456106, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 20:55:06.983150 22748 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:55:06.983948 22748 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:55:06.984745 22634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:06.967450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43065 (local address 127.0.0.1:45890)
0926 20:55:06.967540 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:06.967551 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:06.967589 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:06.969618 (+ 2029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:06.969619 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:06.969789 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:55:06.969854 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:06.969856 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:06.975379 (+ 5523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:06.975380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:06.976236 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:06.976239 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:06.976739 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:06.976740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:06.976795 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:06.977257 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:55:06.977262 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:55:06.978937 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:55:06.983116 (+ 4179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:06.983117 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:06.983118 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:06.983375 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:06.983795 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:06.983796 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:06.983796 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:06.983861 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:06.984375 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:55:06.984376 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:55:06.984461 (+ 85us) client_negotiation.cc:769] Sending connection context
0926 20:55:06.984526 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 20:55:06.984633 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260926 20:55:06.985340 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:06.967673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45890 (local address 127.0.98.60:43065)
0926 20:55:06.969353 (+ 1680us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:06.969355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:06.969363 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:55:06.969383 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:06.969383 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:06.969393 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:06.969447 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:06.974492 (+ 5045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:06.975280 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:06.976330 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:06.976631 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:06.976689 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:06.979016 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:06.979017 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:06.979018 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:06.979025 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:06.982677 (+ 3652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:06.983487 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:06.983489 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:06.983489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:06.983620 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:06.983954 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:06.983955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:06.983955 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:06.984229 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:06.984862 (+ 633us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:06.984990 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 20:55:06.985139 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1665,"thread_start_us":111,"threads_started":1}
I20260926 20:55:06.987198 22631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ec66806433a47b3abd7f75f51703119: Bootstrap starting.
I20260926 20:55:06.988066 22631 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ec66806433a47b3abd7f75f51703119: Neither blocks nor log segments found. Creating new log.
I20260926 20:55:06.988409 22631 log.cc:824] T 00000000000000000000000000000000 P 5ec66806433a47b3abd7f75f51703119: Log is configured to *not* fsync() on all Append() calls
I20260926 20:55:06.991606 22748 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87c8dfe59b704e0aafc579e2b3b9d066: Generated new TSK 0
I20260926 20:55:06.992089 22748 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:55:06.996533 22631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ec66806433a47b3abd7f75f51703119: No bootstrap required, opened a new log
I20260926 20:55:06.998818 22631 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ec66806433a47b3abd7f75f51703119 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c8dfe59b704e0aafc579e2b3b9d066" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39883 } } peers { permanent_uuid: "5ec66806433a47b3abd7f75f51703119" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39101 } } peers { permanent_uuid: "c4fbd2d28065430494041ef76250898e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43065 } }
I20260926 20:55:06.999017 22631 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ec66806433a47b3abd7f75f51703119 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ec66806433a47b3abd7f75f51703119, State: Initialized, Role: FOLLOWER
I20260926 20:55:06.999115 22631 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ec66806433a47b3abd7f75f51703119 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c8dfe59b704e0aafc579e2b3b9d066" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39883 } } peers { permanent_uuid: "5ec66806433a47b3abd7f75f51703119" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39101 } } peers { permanent_uuid: "c4fbd2d28065430494041ef76250898e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43065 } }
I20260926 20:55:06.999495 22631 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ec66806433a47b3abd7f75f51703119 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:55:07.001327 22604 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ec66806433a47b3abd7f75f51703119 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:55:07.002379 22604 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5ec66806433a47b3abd7f75f51703119 [term 1 FOLLOWER]: Refusing update from remote peer 87c8dfe59b704e0aafc579e2b3b9d066: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:55:07.002653 22726 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 87c8dfe59b704e0aafc579e2b3b9d066 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ec66806433a47b3abd7f75f51703119" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 20:55:07.002938 22767 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5ec66806433a47b3abd7f75f51703119: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:55:07.002995 22767 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:55:07.003021 22605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "87c8dfe59b704e0aafc579e2b3b9d066" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5ec66806433a47b3abd7f75f51703119"
I20260926 20:55:07.004472 22605 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5ec66806433a47b3abd7f75f51703119 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87c8dfe59b704e0aafc579e2b3b9d066 in term 1.
I20260926 20:55:07.006927 22756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ec66806433a47b3abd7f75f51703119 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87c8dfe59b704e0aafc579e2b3b9d066" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c8dfe59b704e0aafc579e2b3b9d066" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39883 } } peers { permanent_uuid: "5ec66806433a47b3abd7f75f51703119" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39101 } } peers { permanent_uuid: "c4fbd2d28065430494041ef76250898e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43065 } } }
I20260926 20:55:07.007035 22756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ec66806433a47b3abd7f75f51703119 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:55:07.007081 22756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ec66806433a47b3abd7f75f51703119 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87c8dfe59b704e0aafc579e2b3b9d066. Latest consensus state: current_term: 1 leader_uuid: "87c8dfe59b704e0aafc579e2b3b9d066" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c8dfe59b704e0aafc579e2b3b9d066" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39883 } } peers { permanent_uuid: "5ec66806433a47b3abd7f75f51703119" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39101 } } peers { permanent_uuid: "c4fbd2d28065430494041ef76250898e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43065 } } }
I20260926 20:55:07.007145 22756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ec66806433a47b3abd7f75f51703119 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:55:07.007174 22756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ec66806433a47b3abd7f75f51703119 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "87c8dfe59b704e0aafc579e2b3b9d066" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c8dfe59b704e0aafc579e2b3b9d066" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39883 } } peers { permanent_uuid: "5ec66806433a47b3abd7f75f51703119" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39101 } } peers { permanent_uuid: "c4fbd2d28065430494041ef76250898e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43065 } } }
I20260926 20:55:07.007216 22756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ec66806433a47b3abd7f75f51703119 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 20:55:07.027619 22560 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:55:07.025933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60710 (local address 127.0.98.62:39883)
0926 20:55:07.025981 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:07.025983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:07.026911 (+ 928us) server_negotiation.cc:407] Connection header received
0926 20:55:07.027026 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:07.027028 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:07.027039 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:07.027070 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:07.027546 (+ 476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60710: BlockingRecv error: recv got EOF from 127.0.0.1:60710 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 20:55:07.028584 22710 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:55:07.026296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45896 (local address 127.0.98.60:43065)
0926 20:55:07.026336 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:07.026338 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:07.028063 (+ 1725us) server_negotiation.cc:407] Connection header received
0926 20:55:07.028160 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:07.028161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:07.028172 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:07.028201 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:07.028519 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45896: BlockingRecv error: recv got EOF from 127.0.0.1:45896 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 20:55:07.029441 22636 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:55:07.026633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40644 (local address 127.0.98.61:39101)
0926 20:55:07.026672 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:07.026674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:07.028933 (+ 2259us) server_negotiation.cc:407] Connection header received
0926 20:55:07.029033 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:07.029034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:07.029044 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:07.029073 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:07.029364 (+ 291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40644: BlockingRecv error: recv got EOF from 127.0.0.1:40644 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 20:55:07.032012 22636 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:55:07.031159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40650 (local address 127.0.98.61:39101)
0926 20:55:07.031192 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:07.031194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:07.031431 (+ 237us) server_negotiation.cc:407] Connection header received
0926 20:55:07.031549 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:07.031551 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:07.031567 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:07.031596 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:07.031969 (+ 373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40650: BlockingRecv error: recv got EOF from 127.0.0.1:40650 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 20:55:07.034978 22636 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:55:07.034264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40666 (local address 127.0.98.61:39101)
0926 20:55:07.034295 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:07.034297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:07.034511 (+ 214us) server_negotiation.cc:407] Connection header received
0926 20:55:07.034606 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:07.034607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:07.034616 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:07.034655 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:07.034937 (+ 282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40666: BlockingRecv error: recv got EOF from 127.0.0.1:40666 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 20:55:07.038894 22636 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:55:07.038157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40682 (local address 127.0.98.61:39101)
0926 20:55:07.038188 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:07.038189 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:07.038389 (+ 200us) server_negotiation.cc:407] Connection header received
0926 20:55:07.038493 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:07.038494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:07.038504 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:07.038538 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:07.038848 (+ 310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40682: BlockingRecv error: recv got EOF from 127.0.0.1:40682 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 20:55:07.044956 22636 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:55:07.044192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40684 (local address 127.0.98.61:39101)
0926 20:55:07.044226 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:07.044228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:07.044454 (+ 226us) server_negotiation.cc:407] Connection header received
0926 20:55:07.044562 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:07.044563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:07.044572 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:07.044601 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:07.044910 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40684: BlockingRecv error: recv got EOF from 127.0.0.1:40684 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 20:55:07.055089 22636 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:55:07.054256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40698 (local address 127.0.98.61:39101)
0926 20:55:07.054297 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:07.054299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:07.054540 (+ 241us) server_negotiation.cc:407] Connection header received
0926 20:55:07.054639 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:07.054640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:07.054651 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:07.054679 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:07.055021 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40698: BlockingRecv error: recv got EOF from 127.0.0.1:40698 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 20:55:07.073657 22636 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:55:07.072673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40714 (local address 127.0.98.61:39101)
0926 20:55:07.072720 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:07.072722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:07.072987 (+ 265us) server_negotiation.cc:407] Connection header received
0926 20:55:07.073103 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:07.073104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:07.073117 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:07.073149 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:07.073593 (+ 444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40714: BlockingRecv error: recv got EOF from 127.0.0.1:40714 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 20:55:07.108294 22636 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:55:07.107295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40724 (local address 127.0.98.61:39101)
0926 20:55:07.107340 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:07.107343 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:07.107637 (+ 294us) server_negotiation.cc:407] Connection header received
0926 20:55:07.107776 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:07.107778 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:07.107791 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:07.107824 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:07.108234 (+ 410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40724: BlockingRecv error: recv got EOF from 127.0.0.1:40724 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 20:55:07.174860 22636 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:55:07.173943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40726 (local address 127.0.98.61:39101)
0926 20:55:07.173996 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:07.173998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:07.174180 (+ 182us) server_negotiation.cc:407] Connection header received
0926 20:55:07.174220 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:07.174221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:07.174233 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:07.174299 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:07.174784 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40726: BlockingRecv error: recv got EOF from 127.0.0.1:40726 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260926 20:55:07.305001 22636 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:55:07.304146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40734 (local address 127.0.98.61:39101)
0926 20:55:07.304256 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:07.304259 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:07.304374 (+ 115us) server_negotiation.cc:407] Connection header received
0926 20:55:07.304468 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:07.304469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:07.304487 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:07.304552 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:07.304932 (+ 380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40734: BlockingRecv error: recv got EOF from 127.0.0.1:40734 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 20:55:07.540165 22830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87c8dfe59b704e0aafc579e2b3b9d066 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "87c8dfe59b704e0aafc579e2b3b9d066" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c8dfe59b704e0aafc579e2b3b9d066" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39883 } } peers { permanent_uuid: "5ec66806433a47b3abd7f75f51703119" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39101 } } peers { permanent_uuid: "c4fbd2d28065430494041ef76250898e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43065 } } }
I20260926 20:55:07.540345 22830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87c8dfe59b704e0aafc579e2b3b9d066 [sys.catalog]: This master's current role is: LEADER
W20260926 20:55:07.562976 22636 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:55:07.562221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40738 (local address 127.0.98.61:39101)
0926 20:55:07.562294 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:07.562296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:07.562307 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:55:07.562408 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:07.562410 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:07.562429 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:07.562559 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:07.562909 (+ 350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40738: BlockingRecv error: recv got EOF from 127.0.0.1:40738 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 20:55:07.922957 22723 catalog_manager.cc:1259] Loaded cluster ID: 2f9130daa378488cada713b82b6aa6e6
I20260926 20:55:07.923053 22723 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c4fbd2d28065430494041ef76250898e: loading cluster ID for follower catalog manager: success
I20260926 20:55:07.923861 22723 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c4fbd2d28065430494041ef76250898e: acquiring CA information for follower catalog manager: success
I20260926 20:55:07.924134 22723 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c4fbd2d28065430494041ef76250898e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 20:55:08.003443 22767 catalog_manager.cc:1259] Loaded cluster ID: 2f9130daa378488cada713b82b6aa6e6
I20260926 20:55:08.003558 22767 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5ec66806433a47b3abd7f75f51703119: loading cluster ID for follower catalog manager: success
I20260926 20:55:08.004415 22767 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5ec66806433a47b3abd7f75f51703119: acquiring CA information for follower catalog manager: success
I20260926 20:55:08.004758 22767 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5ec66806433a47b3abd7f75f51703119: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 20:55:08.079169 22842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:08.076215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40752 (local address 127.0.98.61:39101)
0926 20:55:08.076338 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:08.076340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:08.076350 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:55:08.076417 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:08.076418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:08.076466 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:08.076537 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:55:08.076708 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:08.077283 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:08.078082 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:08.078298 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:08.078362 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:08.078554 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:55:08.078841 (+ 287us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:55:08.078923 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:08.079004 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 20:55:08.079026 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
I20260926 20:55:08.083714 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:08.081177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45912 (local address 127.0.98.60:43065)
0926 20:55:08.081294 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:08.081296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:08.081313 (+ 17us) server_negotiation.cc:407] Connection header received
0926 20:55:08.081349 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:08.081349 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:08.081386 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:08.081474 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:55:08.081582 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:08.082086 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:08.082888 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:08.083107 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:08.083169 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:08.083329 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:55:08.083438 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:55:08.083498 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:08.083591 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 20:55:08.083643 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":74,"threads_started":1}
I20260926 20:55:08.088384 22842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:08.085735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40754 (local address 127.0.98.61:39101)
0926 20:55:08.085781 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:08.085783 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:08.085793 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:55:08.085875 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:08.085876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:08.085889 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:08.085938 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:55:08.086119 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:08.086684 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:08.087545 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:08.087881 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:08.087923 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:08.088106 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:55:08.088140 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:55:08.088193 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:08.088280 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 20:55:08.088304 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 20:55:08.089012 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:08.085872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45928 (local address 127.0.98.60:43065)
0926 20:55:08.085923 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:08.085925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:08.086056 (+ 131us) server_negotiation.cc:407] Connection header received
0926 20:55:08.086152 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:08.086153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:08.086164 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:08.086220 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:55:08.086417 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:08.087132 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:08.087955 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:08.088244 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:08.088303 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:08.088669 (+ 366us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:55:08.088702 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:55:08.088731 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:08.088849 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 20:55:08.088869 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 20:55:08.089463 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:08.085699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60724 (local address 127.0.98.62:39883)
0926 20:55:08.086017 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:08.086020 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:08.086028 (+ 8us) server_negotiation.cc:407] Connection header received
0926 20:55:08.086071 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:08.086072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:08.086127 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:08.086224 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:55:08.086345 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:08.087133 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:08.088377 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:08.088691 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:08.088757 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:08.088965 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:55:08.089052 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:55:08.089233 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:08.089343 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 20:55:08.089376 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":67,"threads_started":1}
I20260926 20:55:08.095314 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 22488
I20260926 20:55:08.100751 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 22563
I20260926 20:55:08.105695 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7paSjL/build/release/bin/kudu with pid 22638
2026-09-26T20:55:08Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2815 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5654 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (382588 ms total)
[ PASSED ] 54 tests.
[ FAILED ] 2 tests, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
[ FAILED ] SecurityITest.TestEncryptionWithKMSIntegration
2 FAILED TESTS